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Executive Summary 


Introduction 


The Centres for Learning on 
Evaluation and Results (CLEAR) 
initiative is a collaborative global 
partnership, established in 2010 as 
an experimental approach to 
developing country government and 
civil society capacity in monitoring 
and evaluation (M&E) and 
performance management. 


CLEAR's overall strategy is to 
integrate local knowledge and 
experience, on-the-ground support, 
and institutional development with 
global public goods in monitoring 
and evaluation. CLEAR's intended 
outcomes are shown in the sidebar. 


CLEAR Highest Level Outcome 


To contribute to stakeholders in the target regions using evidence in 
making decisions for improved development results 


Higher Level Outcome 
Strengthened context-specific M&E systems and practices 
CLEAR's immediate intended outcomes 


Improved enabling environments and strengthened demand for 
M&E 


Strengthened organisational capacity to produce and use evidence 
Critical mass of professional expertise 

Innovations in M&E 

Source: CLEAR Strategy 2013-2018 


To implement its strategy, CLEAR has regional and global components. The global component is 
intended to generate and share learning about monitoring and evaluation (M&E) across regions 
and the regional component is intended to build supply and demand for M&E capacity. Regional 
Centres (and ultimately a network of such centres) are expected to become self-sustaining within a 
five-year period with income generated from clients. The first Regional Centre was selected in 
December 2010 and the most recent (Brazil) in 2013. In 2013, the overall programme end date 
was extended to 30 December 2018 (for final disbursement of funds), with the possibility of 


extension. 


CLEAR's governance structure encompasses the Board, the Secretariat, and Regional Advisory 
Committees (RAC) established by each of the Regional Centres. The Secretariat is housed in the 
Independent Evaluation Group (IEG) of the World Bank in Washington DC. 


CLEAR is financially supported by 11 partners." It also benefits from oversight functions of World 
Bank departments in the areas of financial management, procurement, and legal agreements and 


assessments. 


Midterm Evaluation (MTE) 


Requirements for a midterm and final evaluation were built into the funding agreements and 
expectations of funders and CLEAR Centres since the inception of the initiative. The midterm 
evaluation also reflects a demand by the CLEAR Board to have an independent assessment of 
CLEAR progress at the mid-point in its lifecycle for the following purposes: 


e Learning for improvements in the rationale, design, management, implementation and 
governance of the CLEAR global Initiative 


e Accountability to the current funders of CLEAR for funds invested in CLEAR 


1 The African Development Bank (AfDB), the Australian Agency for International Development (AusAid), the 
Asian Development Bank (ADB), Belgian Development Cooperation, the Inter-American Development Bank 
(IDB), the Rockefeller Foundation, the Swedish International Development Agency (SIDA), the Swiss Agency 
for Development Cooperation (SDC), the UK Department for International Development (DFID), the World 
Bank (WB), and the Independent Evaluation Group (IEG) 
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e As a public good contributing knowledge on approaches to strengthening evaluation 
capacity in developing countries, designing and managing global initiatives. 


e The evaluation objectives as articulated in the evaluation Terms of Reference emphasise 
the formative nature of the evaluation: 


e Highlight achievements, challenges and lessons to date as a basis for accountability to the 
funders and hosting institutions 


e Make recommendations for improvements in the design, management, governance and 
implementation of the CLEAR Initiative for the remaining period of implementation, with a 
view to Initiative sustainability. This includes identifying the most promising strategies 
and/or alternatives approaches for CLEAR’s success. 


e Produce public good knowledge (lessons, approaches) following the evaluation to inform 
the fields of development evaluation, regional capacity building, institution building, and 
global initiatives. 


The intended users of the evaluation include: the Board of CLEAR, the Secretariat, the five 
operational Regional Centres, and CLEAR clients and collaborators including governments and 
voluntary organisations of professional evaluators (VOPEs). 


The evaluation covers the period from CLEAR inception in 2010 to December 2013 and, where 
relevant, considers developments in the five operational centres up until the time of finalising the 
evaluation report (September 2014). The evaluation considered the effectiveness of the CLEAR 
Board, Secretariat, Regional Centres and their affiliates and Regional Advisory Committees that 
were operational at the time of the evaluation. 


Methodology and limitations 


The evaluation was commissioned by DfID and managed by the Evaluation Task Force (ETF) 
composed of representatives from the CLEAR Board. 


The evaluation was guided by a framework of questions and a data collection matrix that were 
used to assess the performance of CLEAR at global, regional and national levels and to ensure 
consistent collection of data and allow for triangulation of data from different sources. 


Data collection included a literature review and extensive review of CLEAR documents and 
reports; interviews with 270 stakeholders, in person and by email and Skype; site visits to five 
Regional Centres (Anglophone Africa, Francophone Africa, East Asia, South Asia, Latin America) 
and affiliate centres in Ghana, Kenya, and Burkina Faso; and visits to the Secretariat in 
Washington DC. 


Analysis and reporting: The CLEAR Theory of Change and Results Framework were the agreed 
basis for the assessment. The evaluation team used descriptive analysis, content analysis, 
quantitative/statistical analysis, comparative analysis, and network analysis to inform findings. 
Following data collection, the evaluation team presented emerging findings to the Evaluation Task 
Force (ETF) and submitted a draft evaluation report in June 2014. Following feedback from the 
ETF and the CLEAR Board in July 2014, the structure and content of the report were modified and 
a revised report was submitted in September 2014. 


Communication and dissemination: The evaluation team proposed activities to communicate 
and disseminate information on the evaluation to key stakeholder groups (presentation to Board, 
webinars for Regional Centres, learning briefs to be posted on CLEAR website). 


Limitations and mitigation strategies: 


e The original evaluation TOR and methodology did not adequately address the 
experimental nature of CLEAR. This evaluation design shortcoming was identified in a 
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meeting with the CLEAR Board in July 2014. The evaluation team subsequently 
reviewed data and reformulated findings to better reflect the experimental nature of 
CLEAR. 


e The organisational network analysis (ONA) was removed as a line of evidence due to 
the limited and inconsistent survey responses. The survey will be reactivated and the 
resulting analysis will be completed towards the end of 2014. 


e The evaluation TOR indicated an interest in how the cost of CLEAR centre services 
compared to those of other capacity building providers in the respective regions. 
Despite its efforts, the evaluation team found insufficient data to pursue this line of 
evidence. 


Main findings 
Overall 


The CLEAR initiative is highly relevant to the evolving global discourse on results management 
and aid effectiveness, to the M&E needs of governments, and to the priorities of current and 
potential donors. 


CLEAR was intended as an experiment, but this was not consistently reflected in how the initiative 
has been managed to date. CLEAR has not formulated the hypotheses it set out to test, nor has it 
put in place mechanisms to systematically collect or analyse data to test its assumptions. The 
CLEAR Theory of Change and Results Framework provided guidance for the establishment of 
regional centres but were less useful for testing key assumptions, promoting learning within and 
across CLEAR units, and for assessing progress towards envisaged development results. To date, 
neither CLEAR overall, nor each of the regional centres has defined what ‘success’ in development 
terms would look like at global or regional/national levels. 


The CLEAR Secretariat has effectively fulfilled its assigned roles, has provided administrative 
support to the functioning of the initiative, and has provided leadership and guidance for the 
regional centres. The location of the Secretariat in the World Bank’s IEG has both advantages and 
disadvantages; relocating the Secretariat during the current phase of transition would likely pose 
more challenges than potential benefits. 


The CLEAR Board has fulfilled its three assigned roles with varying degrees of success. It provided 
effective leadership on operational matters but less guidance on the questions and issues 
emerging as a consequence of CLEAR’s experimental design, or on longer-term strategic 
decisions on the future of CLEAR. The current Board composition lacks diversity in regional 
representation, experience and expertise, which limits its legitimacy in the eyes of stakeholders. 
Making changes to the composition of the Board (or the addition of a Steering Committee with 
diverse membership) could address some of these issues, but would not automatically solve the 
noted gaps in leadership for guiding an experimental initiative. 


Regional component 


Design: The internal and external contexts of the five reviewed regional centres varied 
considerably; this was not sufficiently accommodated in the programme design and resulted in lost 
learning opportunities. The CLEAR regional centres are in relatively early stages of developing 
their own strategies and do not yet have a clear, appropriate basis for measuring “success” in 
terms of development results. 


Capacities: Regional centres have varying levels of institutional capacity, which in some cases 
limits their potential to make the kinds of contributions envisaged in the CLEAR design. With the 
exception of the Latin America centre, CLEAR regional centres have to date established relatively 
few strategic, longer-term linkages with regional partners and other like-minded institutions. 
Affiliations with their respective host institutions have affected regional centres in different ways, 





e-Pact iii 


CLEAR Midterm Evaluation 





due to structure and administrative requirements, but overall these relationships have enhanced 
the credibility of regional centres and have provided access to potential clients and partners. With 
the exception of the centre in South Asia, progress towards establishing Regional Advisory 
Committees has been slow, depriving most centres of relevant and regionally grounded strategic 
advice. 


Performance: In all regions, CLEAR objectives and activities are considered relevant to the M&E 
needs of government and non-government stakeholders. All five centres have met most of the 
midterm targets, which focused on the establishment of centres and their ability to provide a variety 
of capacity building services for M&E and RBM. Almost all centre achievements to date relate to 
creating favourable conditions that — in the longer term — have the potential to contribute to 
individual actors or organisations producing (and eventually using) more or better evidence, but in 
keeping with the programmes mid-term status, there is limited evidence of their contribution to 
these higher level envisaged results. The likelihood that regional centres and their services will 
continue without CLEAR funding varies — from low in Anglophone Africa to very strong in Latin 
America and South Asia. 


Global component 


CLEAR stakeholders and beneficiaries value many elements of the global learning component, 
such as the Global Forums, CLEAR training modules, and the Secretariat’s support to regional 
centres. However, it is difficult to assess the effectiveness of this component as CLEAR has not yet 
articulated a global component strategy nor its desired results. At midterm, CLEAR units are still 
experimenting with ways and areas of collaboration, and the regional centres have shown varying 
degrees of interest in and capacity to engage in mutual knowledge exchange and related efforts. 
Overall, the global learning component has not yet realised its potential as CLEAR has not 
harvested the knowledge, lessons and evidence emerging from the CLEAR experiment. This is a 
missed opportunity. 


Future directions and recommendations 
Detailed recommendations are presented in Chapter 5 of the report. The following is a summary. 


Recommendation 1: The CLEAR Board should decide if CLEAR is an experiment that is primarily 
intended to generate lessons learned that can inform future phases of CLEAR and other (M&E) 
capacity building initiatives. The Board should approve a set of overarching learning questions that 
CLEAR seeks to answer as well as the types of information that it will monitor at the level of the 
initiative. 

Recommendation 2: For the duration of the CLEAR experiment to 2018, the CLEAR Board should 
limit management and governance changes to those that will help transform the CLEAR 
programme into a strategically poised, learning initiative. The CLEAR Board should approve the 
establishment of an advisory committee and the appointment of a senior advisor to oversee the 
proposed transformation of the CLEAR programme into a learning initiative. 


Recommendation 3: THE CLEAR Board should assume a more strategic role in the future, 
deferring more operational considerations to the CLEAR Secretariat. 


Recommendation 4: The CLEAR Board should not consider any further expansion into new 
regions between now and 2018, and instead focus its efforts and resources on consolidating the 
learning taking place in the current CLEAR centres. 


Recommendation 5: All CLEAR centres should establish and operationalise Regional Advisory 
Committees (RAC) by December 2014. 


Recommendation 6: Each CLEAR centre should develop a centre-specific theory of change (or at 
least elements of such a theory) to clarify key ideas and assumptions on the purpose, priorities, 
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and envisaged results of the centre. Theories of change (ToCs) may differ in their format and level 
of elaboration, depending on existing centre capacity. 


Recommendation 7: Until 2018, the CLEAR Board should approve support for CLEAR centres’ 
growth and development and engagement in the CLEAR initiative as long as their strategies and 
plans are congruent with and add value to CLEAR’s learning and development objectives. 


Recommendation 8: The CLEAR Secretariat should reformulate the global learning component 
and develop an explicit strategy for the Board’s approval. 
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Glossary of Terms 


Affiliate Centre 


Board member 


Demand (for M&E capacity building) 


Demand (for M&E services) 


Demand (for evidence) 


Donors 


Enabling Environment 


Host institution 


A designated organisation or institution that supports a 
Regional Centre in providing CLEAR services within that 
region. At June 2014, CLEAR had four affiliate centres — 
two in AA, and one in each of FA and SA. Affiliate centres 
are sometimes called CLEAR sub-centres. 


Representatives of the donor organisations currently 
funding CLEAR 


Requests for measures suited to strengthen the M&E- 
related awareness, knowledge, skills or individuals or 
organizations; or to strengthen M&E systems 


Actual, latent or potential requests for more and/or better 
(i.e. more rigorous, methodologically sound) M&E to be 
conducted, or M&E systems to be established and used. 
Underlying intentions can include to meet accountability 
requirements, and to generate more/different types of 
credible evidence. 


Requests by decision makers for evidence to assist them 
in decision making. Demand may be actual, latent or 
potential. The reasons for demand may vary by 
stakeholder; some may focus on improving performance, 
others may focus on learning and others on accountability. 


Refers to development banks, bilateral agencies and 
foundations that provide financial support to CLEAR 


The broader system within which individuals and 
organisations function and which facilitates their existence 
and performance. This environment includes, for example, 
existing legal, policy, and institutional frameworks and 
structures, as well as less tangible influences such as 
socio-cultural norms, values, and practices that influence 
the perceptions and behaviours of individuals and 
collectives? 


An organisation (in most cases a university) that has 
agreed to host a CLEAR centre and provide some 
administrative and logistical support 


2 OECD (2011): Perspective Note: The Enabling Environment for Capacity Development. Retrieved on May 
5 from: http://www.oecd.org/development/governance-development/48315248.paf . 
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Innovation Innovation is the introduction of something new — an idea, 
method or device. Typical categories of innovation for 
monitoring and evaluating the performance of public 
policies, programmes or service delivery are: a) 
technological innovations; innovative products, services, 
processes; or innovative interactions and partnerships. In 
this evaluation, the evaluation team considers a product, 
process, service or technology to be an innovation in M&E 
if at least two of the following criteria are met: a) there are 
significant process improvements; b) there is catalytic 
change; and/or c) the innovations are concrete.’ 


Network In the context of this evaluation the use of the term 
network goes beyond its limited sense of a formally 
established and/or managed body (e.g. the network of 
CLEAR Regional Centres) and also refers to patterns of 
(formal and informal) collaboration among individuals and 
organisations across complex systems. 


Other M&E providers and networks National, regional, or international 
individuals/organisations/networks providing some similar 
products/services as Regional Centres 


Program Theory“ An explicit theory or model of how an intervention 
contributes to a set of specific outcomes through a series 
of intermediate results. A full theory includes an 
explanation of how the programs’ activities contribute to 
the results; it does not simply list activities followed by the 
results, with no explanation of how they are linked. One of 
the benefits of articulating program theory is being able to 
systematically review its plausibility and its consistency 
with evidence and utility. 


RC Client Any individual or organisation that has benefited from/used 
CLEAR Regional Centre products and/or services. This 
includes those who have or have not paid fees for these 
services. Some clients may also be current or past 
collaborators. 


RC Collaborator An individual or organisation that has worked 
collaboratively with the RCs, e.g., to plan for, develop or 
deliver products and/or services. Collaborators can be 
current or past CLEAR clients. 


RC Staff Individuals working in one of the Regional Centres. 


3 Discussion Paper Innovations in Monitoring and Evaluating Results (UNDP, 2013). p.4 


4 From Funnell, Sue and Rogers, Patricia. Purposeful Program Theory. Effective Uses of Theories of Change 
and Logic Models Jossey Bass 2011 (p.21). 
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Regional Advisory Committee 


Secretariat Staff 


Theory of Changes 


Thought leaders 


5 Ibid 


Each centre has (or should have) a group of regionally 
based advisors to provide strategic and programmatic 
advice/direction to the centres. 


World Bank IEG staff working (mostly part time) at the 
global CLEAR Secretariat 


A description of the central mechanisms by which change 
comes about for individuals, groups and communities 
targeted by a development intervention. There might be 
different theories of change at different stages of the 
programme or for different groups of people. 


Nationally, regionally, or globally recognised individuals or 
organisations that actively contribute to public 
thinking/discussions on the theory and practice of M&E 
and related capacity development 
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1 Introduction and Background 


1.1 Background: The CLEAR Initiative 


The Centres for Learning on Evaluation and Results (CLEAR) is a global multi-donor initiative that 
began in 2010. It was conceived as an experiment/pilot to test an innovative, entrepreneurial 
approach to developing country government and civil society capacity in monitoring and evaluation 
(M&E). Based on the rationale that regional learning can enhance global learning and vice versa, 
CLEAR’s ultimate goal is that stronger M&E systems and practices will lead to increased use of 
evidence in decision making. 


CLEAR’s strategy is to build local capacity for M&E through regional centres and to share global 
knowledge and capacity building approaches for M&E across regions. To implement its strategy, 
CLEAR is structured at two levels: 


Global level — CLEAR focuses on global public goods through identifying, generating, and sharing 
innovative and internationally benchmarked knowledge and capacity building approaches through 
a global Secretariat that co-ordinates and facilitates exchange and peer-to-peer learning on M&E 
and performance measurement across and between Regional Centres. 


Regional level — Competitively selected Regional Centres (RC) located in institutions based in 
partner countries provide capacity building services to a range of regional stakeholder groups in 
government and civil society. In some regions, RCs are supported by Affiliate Centres that support 
an RC in providing CLEAR services in other countries in the region. At June 2014, there were four 
affiliate centres. 


The CLEAR theory of change (2013) and the results framework (2013) are presented below. 
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Figure 1.1 CLEAR Theory of Change 
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Table 1.1 CLEAR Results Framework 


Key Performance Indicators 





Highest-Level Outcomes to By 2018, 70 per cent of strategic clients and stakeholders surveyed report 
which CLEAR Contributes increased use of evidence in decision making 


Stakeholders use evidence 
in making decisions for 
improved development 
results 


Higher-Level Outcomes to By 2018, an external evaluation commissioned by the Board indicates that 
which CLEAR Contributes centres have contributed to strengthening of M&E systems. 


Strengthened context- By 2018, an external evaluation commissioned by the Board indicates that at 
specific M&E systems and least 70 per cent of CLEAR clients are using the knowledge, skills, or 
practices information they gained to raise evaluation practice 


Regional Learning 


CLEAR’S Outcomes By their third year, centres demonstrate in their work plans the capacity to 
Regional Learning address a range of M&E topics and methodologies (increase from baseline). 


Enhanced enabling By their third year, centres demonstrate in their annual work plan the 
environment and capacity to offer capacity building through a variety of modalities aimed at 
strengthened demand different capacity objectives (increase from baseline). 

Strengthened organisational By their third year, at least 50 per cent of centre projects engage clients from 
capacity to produce and use outside of the centre’s home country. 


evidence By their 3rd year, at least 80 per cent of service clients score the quality of 
Critical mass of professional service as a 4 or higher (on a five-point scale). 
expertise developed By 2018, an external evaluation indicates that the centres are functioning 


Innovation in M&E well with respect to their strategic plans and objectives 


CLEAR Programme-Level By 2012, five centres selected and operational (original target was four). 
Outputs By 2018, an external evaluation indicates that the centres are functioning 
Regional Centres well with respect to their strategic plans and objectives 


Established and Functional gy 2018, centres’ per cent of revenue-generating activities and programs 
increase from baseline (targets will vary centre to centre) 


Global Approach 


Outcomes of Global By 2018, centre directors and staff report that they have been able to apply 
Learning knowledge gained from other Regional Centres s through the CLEAR 
CLEAR global knowledge initiative 
Peer-learning through the By 2018, the regional centres choose to continuing sharing knowledge and 
network expertise through a global network 
By 2018, a survey of strategic clients and stakeholders indicate that at least 
80 per cent recognise the CLEAR global brand as a source of excellence 
and innovation in M&E 


CLEAR Structure and Governance 


CLEAR is a collaborative effort among donors and partner countries. lts Secretariat is housed in 
the Independent Evaluation Group (IEG) of the World Bank. It operates according to the policies 
and business processes of the Global Partnership and Trust Fund Operations department which 
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serves as a liaison between the external and internal clients on policy programme management 
and best practices. The Charter of CLEAR (see Annex J) provides the policy framework within 
which the CLEAR programme operates. 


A study of CLEAR governance (AccountAbility, 2012) provided a visual representation of CLEAR 
structure as shown in Figure 1.2. 


Figure 1.2 CLEAR Organisational Structure® 


Current Structure— Governing Board, Secretariat have oversight of Operations/ 


Administration, RACs provide advice to the regional centers 
ve Governments 
(Aid Departments) 
te | 


Secretariat if Foundations 
(Board of Trustees) 











Regional Center 
A 


Accountability 





Regional Advisory Committee 


U Tier 1 Stakeholders 





Source: AccountAbility (2012). Enhancing the Governance System of the CLEAR Initiative: Findings and 
Recommendations Report, p 11. 


À AccountAbility. (November 2012). Enhancing the Governance System of the CLEAR Imitative: Findings and 
Recommendations Report, p 11 
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CLEAR is governed by a Board of 13 members representing 11 donor agencies.’ It is 
operationalised through the Secretariat and Regional Centres and affiliate centres responsible for 
CLEAR programming in each region (see Table 1.2). 


Table 1.2 © CLEAR Regional Centres, Host Institutions, and Affiliate Centres 


CLEAR Regional Centres | Host Institution = Affiliate Centres 


Anglophone Africa (AA) University of Witwatersrand (Wits) Ghana Institute of Management and 
Public and Administration (GIMPA) 


Kenya School of Government (KSG) 


Francophone Africa (FA) Centre Africain d'Études 2ie — International Institute for Water 
Supérieures en Gestion (CESAG), and Environmental Engineering, 
Senegal Burkina Faso 

South Asia (SA) Abdul Latif Jameel Poverty Action Centre for Economic Research 
Lab (J-PAL) at the Institute for (CERP), Pakistan 


Financial Management and 
Research (IFMR), India 


Latin America (LA)8 Centro de Investigación y Docencia 
Económicas (CIDE), Mexico 


East Asia (EA) The Asia Pacific Finance and 
Development Centre (AFDC), China 
based at the Shanghai National 
Accounting Institute 


Brazil and Lusophone Africa Centre for Applied Economics (C- 
(added in late 2013) MICRO), Escola de Economia de 
Sao Paulo, Brazil 


CLEAR grants to centres are administered through the World Bank’s administrative, procurement, 
financial, legal and related arrangements. The grants use funds from the multi-donor trust fund 
established for CLEAR. The grant agreement or MOU for each regional centre includes the 
requirement to establish a Regional Advisory Committee (RAC), the role of which is to provide 
“advice and guidance regarding the Centre’s strategies and work programmes based on their 
expertise and knowledge of regional issues with respect to M&E.” ° 


Timeframe 


CLEAR was designed as a five-year programme. However, because the CLEAR centres were 
established at different times, their respective CLEAR grants have different end dates. The first 
couple of centres (Anglophone Africa and South Asia) were selected towards the end of 2010 and 
the most recent (Brazil and Lusophone Africa) in 2013. Also in 2013, the overall programme end 
date was extended to 30 December 2018 (for final disbursement of funds), with the possibility of 
extension. 





7 The African Development Bank (AfDB), the Australian Agency for International Development (AusAid), the 
Asian Development Bank (ADB), Belgian Development Cooperation, the Inter-American Development Bank 
(IDB), the Rockefeller Foundation, the Swedish International Development Agency (SIDA), the Swiss Agency 
for Development Cooperation (SDC), the UK Department for International Development (DFID), the World 
Bank (WB), and the Independent Evaluation Group (IEG). 


8 Spanish speaking. 
2 CLEAR Strategy 2013-2015, p.13. 
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CLEAR Funding and Support 


The sources of CLEAR funding and other support are shown in Table 1.3. The programme also 
relies on oversight functions of World Bank departments in the areas of financial management, 
procurement, and legal agreements and assessments. 


Table 1.3 CLEAR Funding Sources and Contributions 


Donors Contributions for 2010-2018 USD 15 million, fluctuating based on 
exchange rates 


(USD 12.5 million received as of 
December 2013) 


World Bank IEG Budget for administrative and global Approximately USD 790,000 from 
programming work led by the Secretariat inception to June 2013 (excluding 
staff time and overheads) 


World Bank IEG In-kind and overhead support through the Services are not charged to the trust 
CLEAR Secretariat staffed by World Bank  fund/programme 
professional and administrative 


employees* 
World Bank Africa Institutional development grant (IDF) for USD 1 million 
Region the Francophone Africa Centre 


“Additionally, the programme relies on oversight functions of World Bank departments in the areas of financial 
management, procurement, and legal agreements and assessments 


1.2 Evaluation Purposes and Objectives 


Requirements for a midterm and final evaluation were built into the funding agreements and 
expectations of funders and CLEAR Centres since the inception of the initiative. The midterm 
evaluation also reflects a demand by the CLEAR Board to have an independent assessment of 
CLEAR progress at the mid-point in its lifecycle for the following purposes: 


e Learning for improvements in the rationale, design, management, implementation and 
governance of the CLEAR global Initiative 


e Accountability to the current funders of CLEAR for funds invested in CLEAR 


e As a public good contributing knowledge on approaches to strengthening evaluation 
capacity in developing countries, designing and managing global initiatives. 


The evaluation objectives as articulated in the evaluation Terms of Reference (see Annex A) 
emphasise the formative nature of the evaluation: 


e Highlight achievements, challenges and lessons to date as a basis for accountability to 
the funders and hosting institutions 


e Make recommendations for improvements in the design, management, governance and 
implementation of the CLEAR Initiative for the remaining period of implementation, with 
a view to Initiative sustainability. This includes identifying the most promising strategies 
and/or alternatives approaches for CLEAR’s success 


e Produce public good knowledge (lessons, approaches) following the evaluation to 
inform the fields of development evaluation, regional capacity building, institution 
building, and global initiatives. 
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The intended users and uses of the evaluation include: 


e The Board of CLEAR - through the evaluation findings related to governance and 
strategic oversight of the direction, performance and possible expansion of the CLEAR 
Initiative 


e the CLEAR Secretariat at the World Bank — in terms of findings on the management of 
the CLEAR Initiative and their dialogue with Centres 


e Each of the CLEAR Centres (Asia, Latin America, Africa) regarding the performance of 
on-going operations and development and lesson for improvement 


e CLEAR Donors (banks, bilateral agencies, foundations) — in terms of informing 
decisions on continued financing 


e Private sector actors, governments and voluntary organisations of professional 
evaluators (VOPEs) - on lessons in strengthening demand and supply of evaluation 
capacity. 


Scope of the Evaluation 


The evaluation covers the period from CLEAR inception in 2010 to December 2013 and, where 
relevant, considers developments up until the time of report revisions (August 2014). The 
evaluation covers all Regional Centres and their affiliates that were operational at the time of the 
evaluation, '° the CLEAR Secretariat, Board, and Regional Advisory Committees. 


1.3 Evaluation Methodology 


The following is a summary of the evaluation methodology. The full methodology is presented in 
Annex B. 


Evaluation Approach and Framework 


In order to provide information not only on CLEAR programme performance but also to help 
explain how and why the programme potentially is or is not achieving intended results, the 
evaluation adopted a theory-driven approach. This involved reviewing CLEAR effectiveness in the 
context of its explicit theory of change, as well as our understanding of its implicit theory of change, 
and taking into account that CLEAR was designed as an innovative, experimental initiative that is 
still evolving. In addition, the team developed a draft alternative Theory of Change (see Annex E) 
to illustrate the types of assumptions that are not yet captured in the CLEAR ToC. 


The evaluation was guided by the questions in the Terms of Reference (Annex A). A detailed 
framework of questions and a data collection matrix (Annex C and D) were used to assess the 
performance of CLEAR at global, regional and national levels and to ensure consistent collection of 
data and allow for triangulation of data from different sources. 


Evaluation Management and Process 


The evaluation was commissioned by DfID and managed by the Evaluation Task Force (ETF). The 
ETF established an Evaluation Committee and worked with the Universalia evaluation team 
throughout the evaluation (providing guidance, reviewing reports and presentations, providing 
feedback, resolving challenges). The evaluation process consisted of four phases described below. 


10 Since the Regional Centre in Brazil is very new, the evaluation team did not conduct a site visit to this 
centre, but instead consulted with selected stakeholders by telephone. Issues covered with these 
stakeholders focused on the dimension of CLEAR relevance in the Brazilian context, as well as on CLEAR’s 
administrative structures and processes. 
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Phase | — Inception 


During the Inception Phase the evaluation team met with the CLEAR Board to clarify the evaluation 
purpose and objectives and methodology, reviewed relevant documents, and interviewed more 
than 40 CLEAR stakeholders to obtain their views on how the evaluation could add most value to 
CLEAR and their organisations. The team attended the global forum in Mexico in November 2013 
where it interviewed stakeholders and participated in a CLEAR workshop on financial 
sustainability. The team consulted external experts to obtain their insights on the design of the 
network analysis component. The heads of five Regional Centres were contacted to confirm the 
timing of the field visits and to outline expectations and support requirements during the data 
collection phase. All persons interviewed were asked to suggest names of key persons who should 
be consulted during the evaluation. The list was updated over the course of the evaluation as new 
informants were identified. An Inception Report was submitted to the ETF in December 2013 and 
approved in February 2014. 


Phase Il - Data Collection and Field Visits 


Data collection took place between November 2013 and August 2014. The evaluation team, 
supported by regional consultants, collected data to inform the organisational assessment of each 
Regional Centre and to assess the performance of the CLEAR programme in each region. The 
evaluation team used a mix of quantitative and qualitative data collection methods as described 
below. 


Document Review: The evaluation team reviewed literature on capacity building and various 
types of documents, including minutes of Board meetings (made available by the ETF, Secretariat 
and Board), strategy documents, reports commissioned by CLEAR, DfID reports, and documents 
specific to the Regional Centres, 

etc. The list of documents reviewed 


is presented in Annex G. Given that this is a midterm evaluation, our focus was on learning 
Interviews with Key about the types of benefits and challenges/ issues that CLEAR staff, 
Stakeholders: The evaluation clients and partners are raising, rather than on measuring impact. 

E We therefore used semi-structured interviews rather than a written 
consulted 273 stakeholders at ; survey. This allowed for qualitative discussions with stakeholders 
global and country level to obtain but did not yield quantifiable results. 
their perspectives. The list of 


: In th rt wh fer to ‘several stakeholders mentioned x’ 
stakeholders consulted is presented NAS Peper WAEN WE TOET eee ato io Dolo 


this means that at least three individuals in a region or at global level 


In Annex F, which also provides raised a particular issue or had a similar point of view, independently 
information on the types of from one another. We took this as an indication that the issue or 
organizations that the respective viewpoint was important enough to be brought up for CLEAR 
individuals represent, and their consideration. Where applicable we indicate if a particular view was 
relationship with CLEAR. The expressed only in a certain region, or only by certain types of 


stakeholders (e.g. CLEAR clients but not staff). When we say ‘most 
stakeholders said x’, this indicates that no stakeholders expressed 
distinct differing viewpoints from the one reported on. 


processes used to identify 
stakeholders are provided in Annex 
B. See also sidebar. 


Site Visits to CLEAR Regional Centres and Affiliate Centres: The evaluation team, supported 
by regional consultants, conducted visits to five Regional Centres (Anglophone Africa, 
Francophone Africa, East Asia, South Asia, Latin America) and to affiliate centres in Ghana, 
Kenya, and Burkina Faso, and where feasible visited CLEAR-funded initiatives. 


Site visit to CLEAR Secretariat: The evaluation team leader and other core team members 
collected data during site visits to Washington, DC and also through telephone/Skype and email to 
inform the reviews of the CLEAR Secretariat and of the network. 
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Phase III — Analysis and Reporting 


The CLEAR Theory of Change and Results Framework were the agreed basis for the assessment. 
The evaluation matrix was used to structure data analysis and formulate findings on the basis of 
data collected at three levels: key questions, sub-questions, and indicators. 


The mixed methods provided opportunities for triangulation through the convergence and 
overlapping of different methods and different sources to ensure the reliability of information and to 
increase the quality and credibility of the evaluation findings and conclusions. 


Towards the end of the data collection phase, the evaluation team made a presentation of 
emerging evaluation findings to the Evaluation Task Force to validate findings, resolve issues, and 
inform subsequent data collection as necessary. Following the completion of data collection and 
analysis, the evaluation team prepared a draft evaluation report. 


Following the first round of comments on the draft evaluation report, the evaluation team attended 
a working session with CLEAR Board members in Washington D.C. (July 2014). The purpose of 
the meeting was to ensure shared agreement on key messages deriving from or missing in the 
draft evaluation report, and to ensure a shared understanding of Board members’ needs and 
expectations as regards the structure and foci of the report. Following this session, and taking into 
account feedback received from the Evaluation Committee, the report was revised. 


Limitations and mitigation strategies 


Some significant limitations were encountered that adversely affected the evaluation results. These 
are outlined below. 


e The evaluation TOR and resulting evaluation methodology were not fully suited to 
address the experimental nature of CLEAR. This was identified as a shortcoming in the 
evaluation design during the meeting with the CLEAR Board in July 2014. Subsequently 
the evaluation team analysed available data and adjusted the findings to better reflect 
the experimental nature of CLEAR. 


e The evaluation design included an organisational network analysis (ONA) to map and 
measure knowledge sharing between and among key CLEAR stakeholders at global, 
national and regional levels. There were several delays and inconsistencies in 
completion of the ONA survey which limited the utility of the information generated. In 
July 2014 the evaluation team and the CLEAR Board agreed that the ONA would be 
removed as a line of evidence and that the ONA survey would be reactivated with the 
active support of CLEAR RCs. It is anticipated that the resulting analysis will be 
completed towards the end of 2014. 


e The evaluation TOR had indicated an interest in how the cost of CLEAR centre services 
compared to those of other capacity building providers in the respective regions. 
Despite its efforts, the evaluation team was able to elicit only a few examples and 
decided not to include this line of evidence in the report. 


1.4 Report Overview 


In its comments on the draft evaluation report the Evaluation Committee requested that i) more 
information on progress made at midterm be brought into the body of the main report, and ii) that 
the report be restructured to provide more information on progress made by each centre and 
relatively less emphasis on CLEAR governance and organisational arrangements. These requests 
are reflected in the structure (and length) of the revised evaluation report, and the omission of the 
separate volume summarising CLEAR progress against its midterm indicators. 


This document presents evaluation findings on: 
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Chapter 2: CLEAR - Overall programme. This chapter presents evaluation findings on 
the relevance of CLEAR in its various contexts, strengths and weaknesses of the 
CLEAR design, and CLEAR governance and leadership with a focus on the roles and 
performance of the Board and Secretariat. 


Chapter 3: CLEAR — Regional Component contains a summary of key issues 
emerging across the five reviewed centres. It then presents findings on the performance 
of each of the five established regional centres in relation to the criteria of 
relevance/design, effectiveness, efficiency, and sustainability. 


Chapter 4: CLEAR - Global Learning Component. This chapter summarises 
evaluation findings regarding the design and effectiveness of the global learning 
component of CLEAR. 


Chapter 5: Future Directions provides concluding remarks and presents forward- 
looking recommendations. 


The annexes noted throughout the report are presented in a separate document. 
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2 CLEAR - Overall Programme 


This chapter presents evaluation findings on the relevance of CLEAR in its various contexts, the 
strengths and weaknesses of the CLEAR design, and CLEAR governance with a focus on the 
roles and performance of the Board and Secretariat. 


2.1 CLEAR's Relevance 


Finding 1: CLEAR has been and remains relevant in light of the global discourse on 
results management and aid effectiveness. 


Since the mid-1990s, results management'' has become increasingly common in both public and 
not-for profit sectors. “The intent behind this movement is to deliberately measure what results — 
i.e., outputs and outcomes — are brought about by policies, programs and services, and to use that 
information to help better manage public funds and to better report on the use of those funds.”*? 


In parallel, increasing emphasis has been placed by various governments on promoting the 
rational and systematic use of evidence to inform policy.*? The core idea underlying the evidence 
movement" is to replace ideologically-driven policies with decision making that is informed by 
systematic evidence and rational analysis as the basis for producing better outcomes.** This idea 
also underlies the increasing attention paid by various actors to results management and M&E in 
the context of international development and poverty reduction, based on the assumption that 
“better utilisation of evidence in policy and practice can help save lives, reduce poverty and 
improve development performance in developing countries.”'® 


Many global and regional?” organisations and bodies have expressed agreement with and support 
for the development effectiveness agenda and its call for increased mutual accountability and 
transparency, as well as for the related push for strengthening existing capacity for M&E and RBM 
in the global South.** 





11 The evaluation team uses the term results management to cover a variety of terms used in the literature, 
such as results-based management, managing for results, managing for outcomes, outcome-focused 
management, performance-based management and performance management. After J. Mayne (2009) 
Results Management: Can Results Evidence Gain a Foothold in the Public Sector? In: Reiper, O., Leeuw, F. 
and T. Ling (Eds.), 2009. The Evidence Book, Transaction Publishers 


12 John Mayne (2009). 


13 Several authors note that the notion of evidence-based decision making is misleading, as it implies a 
mechanical process without room for judgment or experience. The term ‘evidence-informed’ nuances the 
relationship between evidence and decisions. See, for example, Mayne (2009), and Kirkhart, K. (2000). 
Reconceptualizing evaluation use: An integrated theory of influence. In V. Caracelli & H. Preskill (Eds.), The 
expanding scope of evaluation use. New Directions for Evaluation, No. 88 (pp. 5-24). San Francisco, CA: 
Jossey-Bass. 


14 The evaluation team used the generic term evidence movement to cover movements such as for 
evidence-based management, evidence-based practice, and evidence-based policy and practice. 


15 Sutcliffe and Court (2005). Evidence-Based Policymaking: What is it? How does it work? What relevance 
for Developing Countries? Overseas Development Institute, November 2005. 
http ://www.odi.org.uk/sites/odi.org.uk/files/odi-assets/publications-opinion-files/3683.pdf 


16 Sutcliffe and Court (2005). 


17 Including, for example, the African Union, the Economic and Social Commission for Asia and the Pacific 
(ESCAP), and the Association of Southeast Asian Nations. 


18 For a complete list of organisations, see: http://www.oecd.org/dac/effectiveness/busanadherents.htm 
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As illustrated in the CLEAR Charter (2012) and its Strategic Plan (2013-2018), the rationale for 
CLEAR derived from international 
discussions in the context of the 


Millennium Development Goals 
(MDGs) agreed to in 2000, and “Strengthen our efforts to achieve concrete and sustainable results. 


subsequent agreements on aid This involves better managing for results, monitoring, evaluating and 


: A | communicating progress; as well as scaling up our support, 
effectiveness, in particular the strengthening national capacities and leveraging diverse resources 
Paris Declaration (2005), the Accra | and initiatives in support of development results.” 

Agenda for Action (2008), and the 


Busan Partnership for Effective 
Development (2011), all of which 
promote evidence-informed decision making to enhance development results and emphasise the 
need for measuring results in key development areas as well as mutual accountability for results 
from both donors and developing country governments. 1° 


The Busan Partnership agreement commits its signatories to 


(Busan Partnership agreement, Paragraph 12) 


CLEAR also responded to the acknowledgement of development partners” that despite almost a 
decade of related efforts, significant gaps remain in national capacities for results-based 
management and M&E in terms of both supply and demand. However, a decade is generally 
considered a short time in terms of realising results from capacity development initiatives, and, as 
noted in the current literature, the use of evidence in decision making is not strong anywhere in the 
world, including in developed countries. Individuals involved in the original design of CLEAR 
indicated that the idea for the initiative also derived from deliberations on the affordability and cost- 
efficiency of existing efforts to strengthen national M&E capacities that were largely driven by and 
delivered by actors in the Global North. 


Ongoing international discussions about the Post 2015 Development Agenda re-confirm the desire 
and need for strengthening monitoring and evaluation capacity at all levels in order to ensure that 
development efforts cease to be hindered by a lack of even basic data about social and economic 
circumstances in which people live. “Stronger monitoring and evaluation at all levels, and in all 
processes of development (from planning to implementation) will help guide decision making, update 
priorities and ensure accountability.”2 


Finding 2: CLEAR objectives are congruent with the priorities of current (and many 
potential) donors who are concerned with advancing and accounting for 
development effectiveness. 


All consulted CLEAR donors confirmed the relevance of the initiative’s objectives for their 
organisations due to its potential to help realise objectives of the aid effectiveness agenda. Donor 
agencies have been one of the driving forces behind this agenda and related efforts to promote 
accountability and managing for results. Some current CLEAR donors, in particular the UK 
Department for International Development (DfID), have also been strong promoters of evidence- 
informed policy making in both their own countries and developing countries. For example, the 


18 A common critique in this context is that some donors emphasise accountability while neglecting 
considerations of managing for development results. 


20 For example, the 2012 evaluation of the Paris Declaration: 
http ://www.oecd.org/derec/dacnetwork/48152078.pdf 
21 Bali Communiqué of the High-Level Panel, March 28, 2013. Cited in: United Nations (2013). A New Global 


Partnership: Eradicate Poverty and Transform Economies Through Sustainable Development. The Report of 
the High-Level Panel of Eminent Persons on the Post-2015 Development Agenda. United Nations 


Publications, New York. http://www.un.org/sg/management/pdf/HLP P2015 Report.pdf 

22 See, for example: S. Sutcliffe and J. Court (2005): Evidence Based Policy Making: What is it? How does it 
work? What relevance for developing countries? Overseas Development institute. Retrieved on April 15, 

2014 from: http://www.odi.org.uk/sites/odi.org.uk/files/odi-assets/publications-opinion-files/3683.pdf. 
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DflD-funded project on Building the Capacity to Use Research Evidence (BCURE) is being 
implemented in various countries, including in India where it is conducted in collaboration with 
Abdul Latif Jameel Poverty Action 

Lab (J-PAL) South Asia. 


Many bilateral and multilateral DFID 

donor agencies are under “(...) when times are tight - as they are now - we have an increased 
increasing pressure from their responsibility to make sure we get the best value for every pound 
constituencies to demonstrate that we spend on overseas aid. We've started a new payment by 


invested resources produce results. results’ approach to aid. In future, in certain circumstances, we'll 
This affects how development aid is only be handing over new money once visible results have been 


managed and disbursed, and many acido © 

donor agencies have shown 

increasing interest in reliable ways for assessing and capturing impact level results. 74 DfID, for 
example, has set up an Independent Commission for Aid Impact to look at all aid spending and 
provide information on whether UK aid cash is having a positive impact. See sidebar. 


Finding 3: CLEAR objectives are relevant in light of the interest in and commitment to 
managing for results shown by government and non-government actors in a 
wide range of countries in all regions. Resulting needs and requests for 
capacity building vary widely between countries and among different types of 
actors. 


All countries in which CLEAR 
Regional Centres or affiliate centres 
are located are signatories to the 


“Here, when civil society or media demand accountability from the 
government, what they usually mean is ‘prove that you didn’t steal 
from us’ as opposed to ‘show us results”. But it’s a process, and 


25 rae 

Busan outcome document” and there is definitely movement in people being interested in what the 
other regional agreements pushing government does and achieves with the available resources.” 

for results-orientation and CLEAR national partner, South Asia 


accountability in the context of 

development results. At the same time, the extent to which this commitment is reflected in national 
or sub-national laws and policies, institutions and their resource allocations, and in regulations and 
practices varies considerably. Similarly, countries differ in the extent to which civil society 
organisations and other actors (donors in particular) demand government accountability and 
transparency, and how that affects demand for and pressure on governments to produce and use 
evidence to demonstrate development results (see example in sidebar). Further details on the 
relevance of the CLEAR centres in their respective regional contexts are included in chapter 3. 





23 DFID website https://www.gov.uk/government/policies/increasing-the-effectiveness-of-uk-aid; retrieved: 
April 15, 2013. 


24 Source: DFID website https://www.gov.uk/government/policies/increasing-the-effectiveness-of-uk-aid , 
retrieved: April 15, 2013. See also the World Bank Development Impact Evaluation (DIME) Initiative 
(http://web.worldbank.org/WBSITE/EXTERNAL/TOPICS/EXTPOVERT Y/EXTISPMA/0,, menuPK:384339~pa 
gePK:162100~piPK:15931 0~theSitePK:384329,00.html#whatrole); as well as the website of the Rockefeller 
Foundation that notes the organisation’s commitment to ‘achieving meaningful and measurable impact’ 
(http:/Awww.rockefellerfoundation.org/our-work/current-work ) 


25 OECD (2012). The Busan Partnership for Effective Development Cooperation 
http://www.oecd.org/dac/effectiveness/Busan%20partnership.pdf) 
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2.2 CLEAR Design 


Finding 4: While CLEAR was originally intended as an experiment or pilot, this has not 
been strongly reflected in how the initiative was designed or managed. 


Consulted current and former Board members involved in developing the original CLEAR concept 
envisaged the initiative as an experiment or pilot that was intended to explore uncharted terrain. 
Linked to that was the expectation of CLEAR being about “letting a thousand flowers bloom’, i.e. 
facilitating the development of context-specific approaches to M&E and RBM capacity 
development that might turn out to be different from each other. 


The comparatively modest overall investment in CLEAR (USD 15 million in Trust Fund 
resources”) corresponds with the intent of creating a pilot, given that such initiatives typically aim 
to reduce the risks connected with introducing and testing innovations by confining the scale of the 


intervention.” The evaluation found that, other than having modest funding, CLEAR was not 
consistently designed, presented, perceived or managed as a pilot or experiment. Outlined below 
are some key elements that are common to most pilot initiatives? and observations on the extent 
to which these have been reflected 

in CLEAR design and 


implementation to date. Testing pilot assumptions 


The purpose of pilot projects is According to UNICEF (2009), periodic reviews (including annual 
generally to test a particular reports) of pilots should not only consider progress in the 

strategy, hypothesis, or model implementation, but also the ongoing relevance and feasibility of the 
intervention with the ultimate hypothesis, model intervention or assumptions that are to be tested. 


intention to replicate or expand the 
initiative or at least its successful elements. As such, pilot initiatives constitute research activities 
with knowledge development as their main goal and basis of their legitimacy. See sidebar. 


While document review and consultations with CLEAR staff at global and centre levels identified a 
number of topics and issues that are of common interest to most CLEAR stakeholders, CLEAR has 
not yet identified one or more agreed upon overarching learning questions that it aims to address 
through its work. This has limited the initiative’s ability to systematically test key assumptions 
underlying its design, and draw insights based on implementation experience to date. 


In a similar vein, deliberations and decisions to expand the CLEAR network into new regions (such 
as Lusophone America or the Pacific) have not been specifically tied to knowledge gained about 
the relevance and feasibility of its associated hypotheses, models or assumptions from 
implementing CLEAR in South Asia, Anglophone Africa or Latin America. Instead, they have been 
based on stakeholder interest (particularly donor interest) in the respective regions. 


The intended nature of the CLEAR initiative as an experiment, and related implications for its 
implementation, has not been consistently or effectively communicated to stakeholders at global 
and regional levels. Strategic documents (e.g. CLEAR Charter), Annual Reports, and the CLEAR 
global website do not describe the initiative in these terms. 





26 In comparison, the Cities Alliance generated financial commitments of more than $ 57 million during its first 
four years. Core funding of the Consultative Group to Assist the Poor (CGAP) during the period 2004-2012, 
ranged between $10 and $13.4 million per year. Annual funding for the (well-established) CGIAR increased 
from $ 531 million in 2008 to $ 860 million in 2012. 


27 Source: H. Vreugdenhil, J.H. Slinger, W. Thissen, and P.A. Ker Rault (2010). Pilot projects in water 
management. Ecology and society 15 (3). 


28 See, for example, UNICEF Programme Policy and Procedure Manual (2009). Also: R. Pawson. and N. 
Tilley (1997). Realistic Evaluation. London, Sage Publications 
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Finally, the conceptualisation of how CLEAR would be monitored and evaluated over time 
emphasises accountability rather than learning; reviewed documents (including the CLEAR 
Charter, reporting templates, as well as CLEAR annual reports) make no reference to the pilot 
nature of the initiative, nor do they require/include lessons or other information required/suited to 
inform the management of an experiment. Accordingly, current Board members and senior World 
Bank staff have varied expectations on the nature of results that CLEAR would and should 
produce, with some expressing the need for CLEAR to produce clearly defined development 
results, and others stressing the benefits inherent in learning deriving from experimentation.?° 


CLEAR theory of change 


Finding 5: CLEAR has not tested its articulated assumptions and also has not articulated 
all of the assumptions on which it is based. 


The original and revised versions of the CLEAR theory of change (ToC) — see sidebar - elaborated 
on the broad objectives outlined in the CLEAR Charter.* The CLEAR Strategy (2013-2018) 
formulates the following eight 

assumptions underlying the revised | Evolution of the CLEAR Theory of Change 


Toc?" SN i , l 
Building on the original concept note (2009), a first version of the 


e Increased use of ToC was developed in 2011 by representatives from the Board, 
evidence in decision regional centres, and Secretariat. 
making will contribute to | At the Ghana Forum (2012), the 2011 ToC was determined to be 
improved development inadequate. At the request of the Board, the current ToC was 
results developed in late 2012/early 2013, alongside an effort to prepare the 
CLEAR Strategy. 


e Strengthened context- 
During the 2013 Tunis Global Forum days were set aside to collect 


specific and high-quality stakeholder input on issues such as CLEAR stakeholders, 


M&E systems outcomes, etc. and these were integrated by the Secretariat into the 
(organisational current ToC. 


processes and 
structures, incentives, etc.) and practices will lead to greater use of evidence by 
decision makers 


e Improved enabling environment (e.g., incentives, legislation, institutional relationships, 
communities of practice) and demand for M&E and PM will support effective use of 
M&E and PM as tools for development 


e Capacity at both levels — organisational and professional — and on both supply and 
demand sides — is needed to strengthen M&E and PM systems and practices 


e Innovations in M&E/PM will drive context-specific, practical, and cost-effective 
approaches 


e Capacity is built through a variety of strategically selected context-specific approaches 
suitable for the clients’ needs and demands 





29 According to Vreugdenhil et al. (2010), one characteristic of pilot initiatives is that any lack of progress 
towards results tends to be more tolerated than in other undertakings in light of the overarching goal to 
generate learning. 


30 1) To select and support regional academic/training centres to provide demand-driven capacity building 
services in RBM and M&E on a regional basis, and 2) to provide a multi-regional forum for exchange of 
ideas, knowledge and information. 


31 From CLEAR Strategy (2013-18) 2013 p.2 
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e Regional institutions are well placed to lead capacity building, because they are able to 
work on a consistent and sustained basis across different clients — government, civil 
society, donors — and identify champions, leaders, professionals who will use M&E 


e Global knowledge and standards customised to suit regional and local contexts are 
central to building capacity. 


While these assumptions are relevant, CLEAR did not establish a method or procedure to test 
these assumptions over time, and the evaluation also noted other important assumptions that were 
not articulated — as discussed below. 


Rationale Assumption(s) ** 


One of the implicit assumptions underlying the CLEAR ToC is that strengthened M&E systems and 
practices will lead stakeholders to demand and use evidence in making decisions for improved 
development results. This assumption raises an important research question in light of existing 
research and evidence pointing towards the many challenges that frequently limit or hinder the use 
of any kind of evidence in decision making. While it is understandable that CLEAR has not yet 
been in a position to provide evidence for the soundness of this assumed link, one would expect 
the ToC to articulate the evidence gaps and related (research) questions. If CLEAR had been 
purposefully managed as a pilot, such gaps might have been flagged earlier in its evolution. 


Causal Link Assumptions” 


Feedback from interviewed global thought leaders suggest that CLEAR’s highest level outcome, to 
influence evidence-based decision making for development results, is what positively sets it apart 
from other (lower aiming) M&E capacity building efforts. Moreover, it has influenced the 
understanding of the regional centres of what they are expected to contribute to.** 


While there may be a sound causal link between CLEAR outcomes and its highest level outcome, 
this assumption has not been tested, nor has CLEAR made efforts to systematically gather such 
evidence. * The current ToC does not explain how CLEAR activities are envisaged to lead to the 
intended outcomes, and what distinctive milestones or progress markers will indicate that progress 
is being made. While it was understood that progress would take different shapes in different 
regions, the absence of region-specific ToCs makes it difficult to assess whether short-term 
achievements to date constitute progress towards the envisaged CLEAR outcomes. This has 
posed challenges to the regional centres that are expected to link their activity or output-level 
achievements to the relatively high level CLEAR outcomes. 


32 Rationale assumptions identify the underlying hypotheses or mechanisms on which an intervention is 
founded, e.g. they elaborate the reasons for why a certain approach or strategy was selected, and to what 
end. The rationale for most interventions is based on some prior evidence and experience. See: John 
Mayne: Using Theories of Change in the Agriculture for Nutrition and Health (A4NH) CGIAR Research 
Program. Unpublished Draft, March 2014. See also: Mayne, J. (2014). Using Theories of Reach to Enhance 
Equity Considerations in Evaluation. Speaking Justice to Power: Ethical and Methodological Challenges for 
Evaluators. K. Forss and M. Marra, Eds, Transaction Publishers. 


33 Ibid. Causal link assumptions relate to how and why the assumed transitions between stages in a change 
process will take place, e.g. what has to happen for these causal linkages to be realised. 


34 As noted, for example, in sections 3.3 and 3.4 on the Latin America and South Asia centres. 


35 In their comments on the draft version of the midterm evaluation report, CLEAR Board members 
suggested that it was inappropriate to put too much emphasis on CLEAR’s highest level outcome, given that 
the core of the work conducted by CLEAR focused on the lower level outcomes outlined in the theory of 
change. While this argument is appropriate in terms of assessing CLEAR effectiveness at midterm, it 
remains relevant in our view to examine the appropriateness of the stated highest level outcome in reviewing 
CLEAR’s design. 
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Furthermore, the CLEAR ToC in its current form implies that the pathways of change for the supply 
and demand sides of M&E (and RBM) are identical, or at least not significantly different from each 
other. This contradicts the rationale underlying the creation of CLEAR, namely the widely accepted 


observation that existing M&E capacity building efforts (that primarily focused on the supply side) 
had not also succeeded in influencing demand for M&E.°” The (generic) ToC developed by the 
evaluation team and presented in Annex E illustrates some of the possible differences in these 


pathways that are not currently 
reflected in the CLEAR ToC. 


Targeted stakeholders 


One of the eight explicit assumptions 
associated with the CLEAR ToC 
highlights the importance of context- 
specific interventions involving a 
variety of different types of 
stakeholders. The current ToC is not 
suited, however, for elaborating on 
the types of contributions that each 
of these stakeholder groups is 
envisaged to make to the broader 
development results, nor does it 
explore how these groups will be 
addressed through CLEAR activities, 
given their very different 
organisational settings. 


Key concepts 


Document review and consultations 
with RC staff indicated that there are 
no agreed upon meanings of some 
key concepts that figure prominently 
in the ToC (e.g. innovation in M&E, 
M&E systems, and enabling 
environment and demand for 
M&E*E). How these concepts are 
interpreted shapes the regional 
centres’ work, it affects how CLEAR 
and its stakeholders interpret what 
‘success’ of their work will look like, 
and what types of actors or 
organisations are likely to constitute 
‘strategic’ partners. See sidebar. 





Innovation 


Within CLEAR the term ‘innovation’ appears to be broadly used to 
refer to ideas, approaches, or tools that are not yet commonly 
known or used in a country or region. In our understanding, 
however, a product, process, service or a technology constitutes an 
innovation in M&E only if it meets at least two of the following 
criteria: 


a) There is significant process improvement, i.e. the innovation 
notably influences how M&E is done 


b) There is catalytic change; i.e. an innovation is not just a cheaper 
or faster way of doing the same thing as before 


c) The innovation is concrete, i.e. while an idea or theory can lead to 
innovations in how M&E is conducted, it does not constitute an 
innovation unless it is applied.*¢ 


Demand 


While the ToC only mentions ‘demand for M&E’, document review 
and stakeholder consultations show that in the context of CLEAR 
‘demand’ is used in at least three different ways: 


- Demand for M&E capacity building services 


- Demand for M&E (e.g. for new/better M&E systems to be 
established and used, or more/better evaluations to be conducted, 
for more/different types of evidence to be collected) 


- Demand for evidence deriving from M&E with the intention to 
use this evidence for accountability, learning, or planning purposes. 


While the three meanings are closely intertwined, they are not 
identical. Similarly, CLEAR documents do not distinguish between 
effective and latent demand (or comment on whether and how this 
distinction is relevant in the context of CLEAR). 


M&E Systems 


M&E systems can be located at various levels (e.g. organisational, 
sectoral, local, sub-national, national). The current ToC does not 
clarify whether all of these are assigned the same relevance, or 
whether the initiative is (primarily) aiming to influence national M&E 
systems. 


36 Discussion Paper Innovations in Monitoring and Evaluating Results (UNDP, 2013). p.4. Also, given that in 
some countries in which CLEAR works there is a need for building very basic awareness of, demand for, and 
skills related to M&E and RBM, the question arises whether fostering innovation in M&E constitutes an 


appropriate and realistic objective. 


37 See, for example Mackay, K. (2007) on specific strategies for strengthening demand for M&E. 
http ://www.worldbank.org/ieg/ecd/docs/How to build ME gov.pdf 


38 The CLEAR Strategy 2013-18 (p.2) notes that the enabling environment entails incentives, legislation, 
institutional relationships, and communities of practice. It does not clarify the relationship between the 


enabling environment and demand for M&E. 
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Conclusion 


In our experience it is not unusual for a complex, global initiative to start out with a fairly broad ToC 
to capture key ideas, which then evolves and matures along with the initiative.%® The evaluation 
identified missing and untested assumptions that are essential in understanding how CLEAR 
intends to achieve its stated outcomes. It further noted the absence of systematic efforts suited to 
test existing assumptions. The resulting lack of clarity limits the usefulness of the current ToC for 
regional centres and for a potential learning community that could benefit from testing CLEAR’s 
assumptions. 


CLEAR Results Framework 


The following observations are informed by our understanding of what constitutes a ‘good’ results 
framework, in particular that it: identifies clear and realistic objectives and related performance 
expectations of what will be accomplished and when; sets a manageable number of relevant 
indicators to track performance; and builds ownership for the framework by ensuring that the RBM 
regime is relevant and useful to managers and flexible enough to accommodate a variety of types 
of programmes. 


Finding 6: The CLEAR results framework provided adequate orientation for the 
establishment of the Regional Centres but has been less useful as a basis for 
assessing their progress towards development outcomes and capturing related 
learning. 


The first version of the CLEAR results framework, developed in September 2011, outlined outputs, 
outcomes, indicators and targets for the initiative. It was slightly revised in May 2013, at the same 


time as the CLEAR theory of 

change. 

The current results framework Linking the CLEAR results framework and Theory of Change 
outlines the envisaged results of All of the current indicators for CLEAR Regional Learning and 
CLEAR by means of proximal Global Approach outcomes focus on the types of changes that are 
outputs and outcomes which are expected in the Regional Centres; none measure the expected “so 
focused almost exclusively on what?” in terms of expected, planned developmental changes in 
measuring whether and to what targeted CLEAR stakeholders’ capacities, practices and so forth. 
extent the Regional Centres have The assumption that operational centres guarantee progress 

been established and are towards the envisaged development results is the conceptual “glue” 
functioning.’ This made sense that implicitly links the results framework with the CLEAR Theory of 


Change. However, this assumption is not based on evidence 
derived from the implementation of similar models and CLEAR has 
not yet put in place systems or processes to systematically test it. 


during the early stages of the 
initiative and provided CLEAR 
donors some assurance that the 
centres were making progress in becoming operational. At the same time, the existing indicators 
and milestones do not explicitly measure or facilitate learning within and across the centres (e.g. in 
relation to common research questions underlying the notion of CLEAR as an experiment). 


Good RBM practices suggest that developmental programmes should have appropriate indicators 
to measure their effectiveness. However, the indicators and milestones in the current results 
framework are of limited use for capturing progress against development results given that they do 


39 This was, for example, the case in CGIAR. 


40 Based on the elements described in: Mayne, J. (2007). Best Practices in Results-Based Management: A 
Review of Experience - A Report for the United Nations Secretariat. New York: UN Secretariat. 


41 The vast majority (11 of 14) of the indicators in the Results Framework focus on the Centre’s capacities or 
performance. 
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not define what “success” will look like in development terms beyond having a network of 
operational centres in place (see 


sidebar).*? 

Consulted centre staff in several Decreasingly meaningful indicator 

regions noted that reporting against | The current target # 3 for measuring CLEAR's regional learning 

the indicators and targets in the outcomes is that 50% of centre projects engage clients from 

results framework is an countries outside the centre’s home country.“ Reporting against this 


accountability exercise that has very | indicator does not capture the (potential) strategic relevance of the 
little, if any, value for organisational different actors that the centre engages with in other countries, or 
learning, and does not inform the adequacy of how it does so. The mere fact that a centre 

een A engages with a large number of clients from other countries says 
specific programme improvements. very little, if anything, about its ability to influence change in these 
In addition, the centres in Latin countries, or the region.” 
America, Anglophone Africa and 
South Asia are aiming to increasingly focus their work. Indicators that focus on expanding the 
centres' portfolios of services and clients act as a disincentive for the centres to act upon what they 
consider to be the most appropriate strategy in their respective context. 


Principles of inclusion and gender equality 


Finding 7: CLEAR’s objectives and services do not explicitly take into account the notions 
of inclusion and gender equality. Nevertheless, at least two centres have been 
making efforts to address related issues in some of their work. 


While principles of inclusion and gender equality are considered good practice in development 
interventions, CLEAR has not defined these as priorities or cross-cutting issues for its work. None 
of CLEAR’s founding or strategic documents comment on whether, to what extent, or how the 
programme is intended to address social development issues around inclusion (or, more broadly, 
diversity) or gender equality. To our knowledge, neither the CLEAR Board nor other CLEAR units 
have noted this as a gap in the CLEAR design or implementation. 


However, elements of gender equality awareness are reflected in CLEAR monitoring tools that 
require the RCs to quantify the number of female and male participants in activities, and related 
information is reported by the centres for some but not all activities undertaken. To date, this 
information is not summarised in CLEAR annual reports, nor is it visibly used to inform planning or 
decision making.“ 





42 Annex L presents the results framework with comments on individual results statements, indicators and 
targets. 


43 Target as formulated in the CLEAR Strategy 2013-2018. 
44 A similar observation was made in the CLEAR Program Review conducted by DfiD in March, 2014. 


45 CLEAR rates ‘low’ based on the guiding criteria suggested by the United Nations Evaluation Group 
(UNEG) for assessing the evaluability of an initiative in view of the extent to which it integrates gender 
equality and human rights into its design and implementation. See: Integrating Human Rights and Gender 
Equality in Evaluation — Towards UNEG Guidance. http://www.uneval.org/document/detail/980 
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None of the CLEAR centres has explicitly formulated specific objectives, targets, or strategies for 
addressing the principles of gender equality and inclusion. Nevertheless, the CLEAR centres in 


South Asia and Latin America have 
made various efforts to address 
these issues in their work (see 
sidebar). SA centre staff also stated 
that their capacity building work 
routinely stresses the importance of 
considering equity in the context of 
what constitutes ‘good’ data 
collection and sampling (i.e. 
emphasising the need to collect 
data from a range of stakeholders 
representing different levels of 
influence, status, gender etc.). 


As part of its series of roundtable discussions on evaluation, the 
South Asia centre, in collaboration with other partners, including 
UN Women, organised a cluster of (so far) three sessions focusing 
on Gender, Evaluation, and Empowerment. * Another area of work 
relevant in this regard is the organisational capacity development 
work conducted by the centre with Nongovernmental organisations 
working on gender equality issues, such as Breakthrough. 


The Latin America centre has entered in an informal agreement 
with the regional UN Women office for Latin America to collaborate 
on publications and capacity building on gender and evaluation. The 
centre co-edited a recently published UN Women guide on 
evaluating the integration of gender perspectives, human rights and 
multiculturalism.*” In addition, CLEAR-LA and UN Women are 
considering integrating a session on gender in the CLEAR-LA 
graduate programme on evaluation. 


46 The sessions addressed: Understanding and measuring Women’s Empowerment; Defining and Measuring 
Women’s Empowerment; Experiences in evaluating policies and projects aimed at ending Violence Against 


Women. 


47 Faundez, A. and Weinstein, M. (2014): Guide for the Evaluation of Programmes and Projects with a 
Gender, Human Rights and Intercultural Perspective. Available under: http://www. clear-la.cide.edu/node/173 
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2.3 CLEAR Governance 


This section reflects on the CLEAR Board and Secretariat in terms of their effectiveness in fulfilling 
their respective roles and on the appropriateness of the CLEAR governance structure. 


CLEAR Board 


Finding 8: The CLEAR Board has fulfilled its three roles with varying degrees of success. 
It has been more effective in operational matters and less effective in providing 
strategic direction for an experimental learning initiative. 


According to the CLEAR Charter, the Board has three broad areas of responsibilities: providing 
strategic directions to the programme; reviewing programme effectiveness and progress; and 
communication and outreach. Interviews with CLEAR Board members suggest that they have a 
good understanding of these three expected roles. The stated roles and responsibilities of the 
CLEAR Board as outlined in its 

Charter comply with good 


governance practice (see sidebar). Good governance practice 


The minutes of Board meetings The functions for the CLEAR Governing Board outlined in its Terms 
illustrate the diversified work of Reference * correspond with four of the six core functions of 
required to get the CLEAR initiative GPP Boards as suggested in the IEG Sourcebook (see Volume III, 
rolling — ranging from budget Annex U). These four are: strategic direction, management 
decisions to programme decisions oversight, stakeholder participation, and audit and evaluation. While 


the two functions risk management and conflict management are not 


ISA discussions -and Some explicitly assigned to the CLEAR Board, it has, in practice, fulfilled 
challenges in scheduling meetings, the later? 


turnover and lack of attendance of 

several donors. The Board provided some strategic guidance and considerable operational help to 
the Secretariat in establishing the five centres. These were time-consuming activities that the 
Board undertook with a high level of collaboration and commitment, according to all those involved. 


Outlined below are observations on the extent to which the Board has fulfilled each of its three 
roles as per its TOR. 


Providing strategic direction for the overall CLEAR initiative 


While the Board provided guidance in establishing RCs and expanding the number of centres in 
different regions, it has provided little, if any, guidance on the short-term strategic questions and 
issues emerging as a consequence of CLEAR’s experimental design or on the longer term 
strategic decisions that would follow the experimental phase (such as clarifying the longer term 
vision for CLEAR, and determining more clearly the respective roles of the Board and Secretariat 
in this regard). This is further discussed under finding 10 below. 


One challenge in providing coherent strategic guidance has been that Board members have 
divergent views on what CLEAR is intended to achieve and what aspects need the most 
attention.°° For example: 


e CLEAR Board views varied on the experimental nature of CLEAR, as noted above. 





48 The TOR are included in the CLEAR Charter. 


49 For example, when an issue was raised regarding the RC in Anglophone Africa in which Board members, 
the Secretariat, and the Centre did not share the same view, the Board meeting minutes show there was 
significant discussion and the Board attended to it. 


50 This had already been noted in the 2012 study on CLEAR governance: AccountAbility. (November 2012). 
Enhancing the Governance System of the CLEAR Initiative: Findings and Recommendations Report. 
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e Some Board members see CLEAR primarily as a capacity building exercise (courses, 
trainees, etc). Others have a higher and wider vision and see CLEAR as a catalytic, 
innovative global initiative contributing to improved development outcomes. Some 
others see it as a potential technical capacity building resource for their regional 
partners. 


e |n terms of the types of stakeholders that should be targeted by regional centres, some 
donors would like to see a stronger focus on engaging with civil society, while others 
encourage a stronger focus on government actors. 


e The extent and direction of a potential future expansion of CLEAR in terms of its 
geographic foci (e.g. AusAid has expressed strong interest in the establishment of a 
new centre in the Pacific region, but it is not clear if other donors share this interest) and 
in terms of the types of countries to be targeted (e.g. donors vary in their views on 
whether and to what extent CLEAR should work with, and in, countries in transition from 
conflict). 


e Board members have different interpretations of some concepts that are integral to the 
CLEAR design, such as: innovation, quality, regional and global results. 


Reviewing programme progress and effectiveness 


Document review and feedback obtained from the Secretariat, Board members, and regional 
centres suggests that the Board is partially fulfilling this role. First, it has set up and is managing 
audits and a midterm evaluation of the initiative. Second, it is monitoring programme progress in 
terms of outputs, activities, and budget.** 


Annual reports have provided the Board with a partial view of the initiative’s progress, in so far as 
they allow verifying if activities have been carried out, if the budget for the period has been 
expensed, and if planned outputs have been produced. The Board (as per the minutes) is 
conscientious in reviewing these aspects, challenges gaps between planned versus actual planned 
activities, and provides a critical eye on outputs produced. 


Nevertheless, annual reports have not provided updates on the programme’s progress toward 
outcomes, and there is no documented evidence of the Board having used the CLEAR Results 
Framework or Theory of Change to drive discussions about such progress. In fact, most consulted 
Board members did not remember what was included in the results framework or, more generally, 
what are the four intended outcomes of CLEAR. As such the CLEAR Board is not yet 
demonstrating good practice in managing for results throughout the chain of results, which is a key 
commitment of the Paris Declaration on Aid Effectiveness. 


Role in communication and outreach: 


There are two aspects to this role: engaging with stakeholders and raising funds. The evaluation 
found evidence (reported through interviews with Board members and in the field) that some 
members of the Board play a key role in both outreach and advocacy for the importance of 
CLEAR. It appears that representatives of larger donors, perhaps due to their organisations’ 
investments in CLEAR, have been more vocal about the programme and have engaged inside 
their organisations to make the programme more visible. Representatives of donor agencies that 
have made relatively modest financial investments in CLEAR report that they have little time to do 
engage in additional fundraising for CLEAR beyond the existing commitments of their respective 
organisation. Other donors, such as DfID and the Rockefeller Foundation have made both 
substantial investments in CLEAR overall, and have also provided additional funds for specific 


51 Please also see finding 4 that explores the extent to which CLEAR has been managed as a 
pilot/experiment. 
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interventions conducted by CLEAR units, e.g. (in case of the Rockefeller Foundation) for efforts 
aimed at building M&E capacities within civil society. 


CLEAR Secretariat 


The Secretariat reports directly to the Board’s Chairperson (Charter, p. 9). Secretariat staff 
members are Independent Evaluation Group (IEG) staff and are assigned to work at CLEAR as 
part of their IEG responsibilities. Secretariat staff report to the World Bank which conducts 
performance reviews with input from the Chair of the CLEAR Board. As noted in the Charter, the 
IEG houses the Secretariat for the first phase of CLEAR, with the option of relocating the 
Secretariat to a partner country in subsequent phases. 


The Secretariat currently has seven staff members who are IEG employees or contract personnel 
and who are assigned to CLEAR for a portion (60-70%) of their work load. CLEAR staff 
workprogram is reviewed by both IEG and the CLEAR Board but hiring is done by IEG. 


Finding 9: To date, the Secretariat has effectively implemented its assigned roles. This 
has been facilitated by its location within the IEG. 


Locating the Secretariat at the IEG was appropriate at start-up as IEG conceived of the idea of 
CLEAR, has a unit that looks at global capacity building for M&E, and has the experience of 
starting and managing IPDET, which was an inspiration for CLEAR. In contrast, one interviewee 
suggested that IEG had taken on a huge reputational risk by hosting CLEAR, as the RCs are 
untested start-ups and there is a risk to IEG’s reputation should they fail. Other stakeholders 
suggested that as an innovative, experimental initiative, CLEAR needs freedom to take risks and 
that being housed within IEG limits these possibilities. In other words, there are differing opinions 
about the pros and cons of where CLEAR should be housed.*? However, the evaluation team and 
almost all consulted stakeholders noted that there are currently no alternative locations for the 
Secretariat that would address the noted limitations while at the same time provide the benefits 
that derive from being housed in the IEG. 


The Secretariat is responsible for the day-to-day administration and management of CLEAR, and 
provides a support system to both the Board and the regional centres. lt also organises, delivers, 
and manages the global learning component of CLEAR (see Chapter 4) and coordinates 
communication and partner relations for CLEAR globally. 


According to document review and consulted CLEAR centre staff, Board members, and clients and 
partners who have been in direct contact with the Secretariat, the unit has carried out its assigned 
responsibilities in timely fashion. Secretariat staff members were commended for their dedication 
and commitment to the initiative, and regional centre staff in all regions unanimously expressed 
appreciation for the various types of tailored support they have received from the Secretariat. 


The Secretariat's contributions have not been limited to administrative responsibilities. As 
illustrated in Annex H, while the CLEAR Board is officially tasked with providing strategic direction 
to the initiative, the Secretariat has provided considerable leadership “from behind” on how to 
operationalise CLEAR. However, as discussed below, expectations regarding the type and extent 
of the Secretariat’s strategic contributions have also been a point of discontent within CLEAR. 


52 Varying views on risks and advantages of housing innovative GPPs within the World Bank are quite 
common. For example, similar concerns were identified in recent evaluations of the Consultative Group to 
Assist the Poor (CGAP). 
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Finding 10: Some shortcomings in CLEAR governance limit its effectiveness in guiding an 
experimental initiative and pose risks for its legitimacy in the eyes of 


stakeholders. 


The structure and composition of 
the CLEAR Board has remained 
relatively stable since the inception 
of CLEAR (see sidebar). The 
CLEAR Charter suggested that the 
initial composition of the Board 
(primarily donors) would be 
reviewed at midterm and that it was 
expected that Board membership 
would be expanded to include 
senior government officials 


Current composition of the CLEAR Board 


The CLEAR Charter stipulated that initially the Board would 
comprise: representatives of donors contributing USD1 million or 
more per year, one rotating seat for donors contributing less than 
USD1 million, a representative of the Independent Evaluation Group 
(IEG), and the World Bank’s two regional VPUs (for the first two 
years for regional donor representation). Donors contributing less 
than USD1 million per year would decide who among them would be 
their representative on the Board. While in practice all donors were 
invited to join the Board, regardless of the size of their contributions, 
attendance at Board meetings has been much stronger from those 
who contributed more than USD1 million per year. 


representing partner countries from 
each region. 


In 2012, the Board commissioned a governance study, conducted by AccountAbility, which 
reviewed the CLEAR governance structure and compared it to 12 other global partnership 
initiatives. The Board decided to await the results of this midterm evaluation before acting on the 
recommendations of the governance study. 


Data collected during the evaluation (through document and literature review and consultations 
with CLEAR stakeholders) confirm the following key findings also noted in the 2012 governance 
study. 


The current Board composition lacks diversity in representation, experience and expertise. 
This limits its ability to ensure or enhance CLEAR’s legitimacy in the eyes of stakeholders. 
It also presents an image of CLEAR as a top-down initiative that does not reflect the principle of 
country ownership. While donors are key stakeholders, they do not represent CLEAR's other 
stakeholder groups such as government officials, experts, RC host institutions, the RCs and 
country units — all of whom have a stake in CLEAR’s success and sustainability, and who have 
valuable knowledge to provide to decision makers. 


CLEAR governance units have not yet developed a shared, well-articulated vision and 
operational strategy for CLEAR that does justice to the initiative’s experimental nature. The 
2012 governance study noted a considerable lack of shared understanding among CLEAR units of 
their respective roles and responsibilities, particularly those of the Board and Secretariat. 
Consultations with CLEAR staff and Board conducted during this evaluation confirmed this 
finding.** This has resulted not only in prolonged frustration among different CLEAR units about 
each other’s’ performance and/or expectations, but also in notable gaps in leadership that neither 
the Board nor the Secretariat have filled to date. 





58 AccountAbility. (November 2012). Enhancing the Governance System of the CLEAR Imitative: Findings 
and Recommendations Report. Most of the other GPEs that the study used as comparators had changed 
their governance composition as the partnership evolved. In all cases, changes to the governance structure 
meant that the Board became less active in operational matters and identified more specific roles for 
selected sub-committees (e.g. staffing sub-committee or M&E sub-committee). 


54 Approximately two-thirds of consulted Board members felt that the Secretariat should have played a 
stronger strategic leadership role. 
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Given CLEAR’s experimental 
nature, it would have been 
important for those guiding and 
managing the initiative to pay 
attention to and act upon the 
learning taking place as CLEAR 
implementation progressed. 
Examples of various short-term and 
long-term questions that the 
evaluation team thinks required 
attention but that were not 
sufficiently addressed by CLEAR 
governance bodies are outlined in 
the sidebar. 


It is important to note that the 
Secretariat was never formally 
tasked (in its TOR or in directions 
from the Board) to address most of 
these issues, nor was it staffed to 
do so. A review of Board minutes 
indicates that there have been 
exploratory discussions of some 
areas of responsibility and strategy, 
but the Board has never formally 
discussed who should be 
responsible for providing direction in 
the noted areas. 


According to Bezanson and 
Isenman (2012) these types of 
governance shortcomings are very 
common among global/multi- 
stakeholder programs and 
partnerships.°° 





Examples of strategic questions that have not been sufficiently 
addressed to date 


Does the CLEAR design match its purpose and experimental 
nature? 


Are the assumptions outlined in the ToC appropriate? How will 
CLEAR test these assumptions? 


Are there gaps in the ToC that need to be addressed? Will activities 
lead to expected outcomes? 


Are the research/ learning questions clear and appropriate? Which 
need to be changed? Added? 


Are the allocated resources (human, financial, other) sufficient to 
realize CLEAR ambitions? To monitor its performance as an 
experiment? 


Are governance and management approaches in keeping with the 
pilot or experimental nature of CLEAR? 


Are roles and responsibilities of various stakeholders clear and 
appropriate given CLEAR ambitions? 


Are the CLEAR outcomes realistic/appropriate? Are the indicators 
appropriate? Is the use of a logical framework appropriate given 
CLEAR’s experimental nature? 


To what extent are the various contractual arrangements with the 
Regional Centres and the affiliates facilitating or hindering 
implementation of the CLEAR experiment? What changes need to 
be made? 


What criteria (or types of evidence) will determine whether there 
should be a second phase of CLEAR, or how this second phase 
should look like? 


What are the envisaged future roles of regional centres and affiliate 
centres? What can ‘sustainability’ of CLEAR (or of centres) look 
like? 


55 Bezanson, Keith A. and Isenman, Paul: Governance of New Global Partnerships: Challenges, Weaknesses 
and Lessons". Policy Paper 014. October 2012. Centre for Global Development. Retrieved from 
http://www.cgdev.org/files/1426627 file Bezanson Isenman FINAL.pdf. The authors note that among the 





frequent governance shortcomings of partnerships/programmes identified by independent evaluations of 11 
such mechanisms are: i) weakness or absence of strategic direction, accountability mechanisms, monitoring 
and evaluation systems, and management of risk; as well as ii) confusion between the roles of management 


versus governance. 
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2.4 Summary/Conclusions 


The midterm evaluation findings are positive in terms of the relevance of the CLEAR initiative to 
the evolving global discourse on results management and aid effectiveness, as well as in relation 
to the priorities of current and potential donors. 


The evaluation noted, however, that while CLEAR was originally intended as an experiment, this 
has not been consistently reflected in how the initiative has been managed. In particular, CLEAR 
has not yet explicitly formulated the hypotheses it set out to test, nor has it put in place 
mechanisms to systematically collect, analyse, or share information on such findings. This gap is 
also evident in the CLEAR Theory of Change and Results Framework, both of which were revised 
in 2011, demonstrating CLEAR’s willingness to adapt tools based on implementation experience. 
The ToC and Results Framework provided useful guidance for the start-up phase of CLEAR and 
allowed CLEAR stakeholders (including donors) to track progress of the centres in becoming 
operational. Both frameworks have, however, been less useful for testing or elaborating on key 
assumptions underlying the CLEAR design, promoting learning within and across CLEAR units, 
and for assessing progress made by the regional centres towards envisaged development results. 
To date, neither CLEAR overall, nor each of the regional centres have explicitly defined what 
‘success’ of the initiative will look like at global or regional/national levels. 


The CLEAR Secretariat has effectively fulfilled its assigned roles, has provided administrative 
support to the functioning of the initiative, and has provided leadership and guidance for the 
regional centres. While the location of the Secretariat in the World Bank’s IEG has both 
advantages and disadvantages, the evaluation team considers that relocating the Secretariat 
during the current phase of transition would likely pose more challenges than potential benefits. 


The CLEAR Board has fulfilled its three assigned roles with varying degrees of success. While it 
has provided effective leadership on operational matters it has not provided adequate guidance on 
the questions and issues emerging as a consequence of CLEAR’s experimental design, or on 
longer-term strategic decisions on the future of CLEAR beyond the current phase. The evaluation 
also confirmed findings of the 2012 governance review of CLEAR that noted that the current Board 
composition lacks diversity in representation, experience and expertise, which is likely to limit its 
ability to ensure or enhance CLEAR’s legitimacy in the eyes of stakeholders. While this issue could 
be addressed by making changes to the composition of the CLEAR Board or the broader CLEAR 
governance structure (e.g. by adding a Steering Committee with diverse membership, as had been 
suggested in the 2012 governance study), such changes would not automatically also solve the 
noted lack of leadership for guiding an experimental initiative. 
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3 CLEAR - Regional Centre Component 


3.1 Summary 


This section summarises key observations on overarching issues emerging from the parallel 
review of the five existing CLEAR centres. It is structured along the same categories as the 
detailed individual centre profiles that follow (in sections 3.2 to 3.6), but focuses on reoccurring 
themes and overarching observations. 


3.1.1 Design 


Finding 11: While the internal and external contexts of the regional centres varied 
considerably, this was not sufficiently accommodated in the programme design 
and resulted in lost learning opportunities. 


The five CLEAR regional centres varied considerably in terms of their internal and external 
contexts. Examples of variations in their internal contexts include: 


e The dates at which the centres became fully operational vary considerably, from 
late 2011 (Anglophone Africa and South Asia) to late 2012 (Latin America and East 
Asia) to mid-2013 (Francophone Africa).56 This has, at least to some extent, influenced 
the number and diversity of capacity development activities carried out by each centre 
at midterm.57 


e The projected grant amounts for the different centres varied, with East Asia and Latin 
America being allocated considerably smaller amounts than the other regions. The 
grant amounts for all five centres were modest given their complex mandate and 
envisaged regional scope. 


There were also noticeable differences in the centres’ external contexts, as illustrated below: 


e The demand for and supply of (quantity and quality) M&E/RBM services and 
related capacity development opportunities. While all countries in which CLEAR 
regional centres or affiliate centres are located are signatories to the Busan outcome 
document? and other regional agreements pushing for results-orientation, the extent to 
which this commitment is reflected in national or sub-national laws and policies, 
institutions and their resource allocations, and in regulations and practices varied 
considerably. 


e Differences in pre-existing awareness, knowledge and skills in M&E of the majority 
of targeted stakeholders (e.g. strong in most of Latin America, and relatively weak in 
most of Francophone Africa), which in turn had implications for the appropriate starting 
level for centre services 





56 While the CLEAR timeline, as shown, for example, in the CLEAR Strategy (2013-2018) indicates that the 
FA centre became operational in October 2012, consultations with CLEAR staff indicate that this was only 
truly the case in mid-2013 when the first capacity building programming was conducted. 


57 For example, the South Asia centre has carried out more than the East Asia and Francophone Africa ones. 
At the same time, the later established LA centre has carried out more interventions than the AA centre 
despite its earlier start. 


58 At the same time, both of these centres have been able to leverage other resources, including from the 
respective national governments where the centres are located 


59 OECD (2012). The Busan Partnership for Effective Development Cooperation 
http://www.oecd.org/dac/effectiveness/Busan%20partnership.pdf) 
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e In addition, regional differences affected the number and quality of collaborators 
that the centres were able to draw upon for the planning and implementation of 
capacity development interventions. 


All CLEAR regional centres have made deliberate efforts to tailor their products and services to the 
specific needs and priorities of different actors and their national and/or institutional contexts in 
order to make them contextually relevant and useful to intended users. Some centres (SA, LA, and 
AA) have increasingly taken to developing custom made training modules for selected client 
organisations to meet their particular needs. 


However, while in principle, each centre was given permission by CLEAR to adapt the overall 
CLEAR theory of change and results framework to its particular context and capacity, expectations 
(as reflected in identified CLEAR’s key performance indicators), including those related to the 
expansion and diversification of centre services, have applied equally to all five centres regardless 
of the variations in their internal and external contexts. 


Moreover, while the CLEAR design makes repeated reference to the importance of context, in 
practice this has not yet been consistently taken into account in the initiative. On the one hand, the 
CLEAR design (wisely) allocated resources for the CLEAR Secretariat to provide capacity building 
support to the centres (implicitly acknowledging variations in their institutional establishment and 
maturity), and centres were able to decide the kinds of activities that they could engage in. On the 
other hand, and as noted in Chapter 2, the CLEAR design did not include strategies, mechanisms, 
responsibilities or resources to capture and benefit from the learning possibilities presented by 
these contextual diversities. In lieu of focusing on learning, centres instead concentrated on 
delivering services to meet the (perhaps relatively less important) expectations outlined in the 
results framework. ® In light of CLEAR’s intended nature as an experimental initiative, this 
represented lost learning opportunities. 


Finding 12: The CLEAR regional centres are in relatively early stages of developing their 
own strategies and do not yet have a clear, appropriate basis for measuring 
“success”. 


One major implication of the differences in regional contexts is that the notion of ‘success’ means 
different things for each regional centre. While CLEAR centre staff are acutely aware of this, the 
centres have not yet (or only partially) articulated their region-specific visions of ‘success’, and 
there has been no reflection and analysis of differences among centres within CLEAR overall. 
Centres have made efforts to link their activities to the overall CLEAR outcomes, but due to the 
high-level nature of these outcomes and the absence of regionally defined pathways of change 
and progress markers, centre results to date appear to be more fragmented than they may turn out 
to be in the longer term. 


Similarly, the extent to which the five centres have made explicit their context-specific 
programming strategies varies. The AA centre’s draft strategy (2013), while not yet completed or 
implemented, is focused and context specific. Similarly, the SA centre was commended for its work 
in developing a centre-specific results framework to identify envisaged intermediate results. The LA 
centre has developed a region-specific strategic framework, but has not been able to use it to 
monitor or report upon its work given the existing reporting formats within CLEAR. 


The evaluation also noted that while all five centres have made some efforts to engage with 
stakeholders from outside their home countries, centre strategies provide little, and in most cases 
no, information on the expected benefits or envisaged development results of regional 
engagement. Positive developments in this regard are the current efforts of the South Asia centre 


60 For example, the requirement for Centres projects to engage at least 50% of clients from outside their 
home countries or an increase in the % of Centre revenues that are generated by the centre through 
revenue generating activities. 





e-Pact 28 


CLEAR Midterm Evaluation 





to develop an explicit regional strategy, the draft strategy developed by the Anglophone Africa 
centre that articulates key elements of the centre’s future engagement beyond South Africa, and 
ongoing efforts by the LA centre to further define its niche in the region. 


3.1.2 Centre capacities 


Finding 13: CLEAR regional centres continue to have very mixed institutional capacities to 
participate effectively in the initiative. This limits their potential to make the 
kinds of contributions envisaged in the overall CLEAR design. 


From the outset, there were vast differences in regional centres: some were created within 
institutions with established track records in M&E capacity building (SA, EA and LA), while others 
were launched in institutions with a modest profile in M&E capacity building (AA and FA). It is 
therefore not surprising that it has taken time for the AA and FA centres to develop the managerial 
and professional capacities needed to engage in the CLEAR initiative in their regions. These two 
centres have been challenged by and continue to face considerable gaps in both professional and 
managerial capacities. Similarly, the centre in East Asia continues to face limitations in its 
professional capacity. 


Finding 14: With the notable exception of the LA centre, CLEAR regional centres have 
placed modest to little emphasis on nurturing linkages with other like-minded 
institutions. 


All CLEAR centres have engaged in partnerships with diverse types of actors within their home 
countries and regions. However, the breadth and diversity of partnerships established to date 
varies between centres, with the Latin America centre having put the most pronounced emphasis 
on working with and through various partners. None of the five centres has developed a network of 
collaborators that is sufficient to meet all requests for capacity development support that it 
receives, and there is room for further clarifying and, in some cases, expanding the roles that 
(potential) strategic partners in the region will play in complementing a centre’s internal capacity. 
The AA centre’s draft strategy includes promising efforts in this regard, as do strategic 
deliberations included in strategic documents of the LA centre. 


Finding 15: The affiliation with their respective host institution has, overall, been beneficial 
for the five regional centres. There is room for further strengthening the 
collaboration between the centres and host institutions, especially in 
Anglophone and Francophone Africa. 


In all regional centres, the reputational implications of being affiliated with their host institutions 
have generally been positive in terms of (potentially or actually) enhancing their credibility and in 
opening doors to potential clients and partners. At the same time, as shown in the sidebar below, 
the affiliation presented both benefits as well as moderate reputational challenges. In several 
cases (AA, SA) the respective host institutions were instrumental in keeping centre activities going 
when there were delays in the receipt of the CLEAR grant. 


The centres were affected in different ways by the structures and administrative processes within 
their host institutions. While in Latin America, South Asia, and East Asia these effects were largely 
beneficial, they posed challenges in Anglophone and Francophone Africa. As described in section 
3.2, the AA centre in particular was also adversely affected by notable gaps in strategic support 
from its host institution. 
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Regional centres and their host institutions 


In East Asia, AFDC’s affiliation with the government of China, especially the Ministry of Finance, enhances the 
centre’s ability to reach out to other government departments, but also limits its ability to engage with non- 
government actors.*! 


In South Asia, the CLEAR centre benefits considerably from the strong reputation enjoyed by J-PAL related to 
its expertise in Randomised Control Trials. However, this has also made it challenging for the centre to step out 
of J-PAL’s shadow and create its own reputation based on a mandate that includes a wider range of 
methodological approaches. This has been even more difficult as most CLEAR staff are also known among 
stakeholders as J-PAL staff. 


In Anglophone Africa, Wits’s long standing association with the ANC government meant that it had the 
necessary strong trust and relationships to engage meaningfully on good governance matters, including the 
use of M&E in decision making. However, the university's governance and administrative requirements took a 
toll on the centre given its modest institutional capacities. 


In Latin America the association with CIDE has positively influenced the reputation and credibility of the centre 
in the eyes of potential clients, especially within the Mexican public sector. Since centre establishment CIDE 
has provided considerable financial, professional and administrative support to the centre. 


In Francophone Africa the CLEAR centre has benefited from CESAG’s positive reputation as a successful 
institution involved in educating future policy and decision makers especially in the context of the Senegalese 
government. The centre has, however, faced challenges due to the cumbersome administrative and financial 
management systems of CESAG, and has not yet strongly benefited from access to its host institutions’ faculty 
members. 


Their affiliation with the World Bank has, overall, benefitted CLEAR centres in both reputational as 
well as practical terms. The latter was constituted, for example, by on-the-ground contacts 
facilitated by the local WB team, and/or the centre having access to local WB funds for supporting 
selected clients through tailored capacity development activities. 


Finding 16: With exception of South Asia, progress towards establishing Regional 
Advisory Committees has been slow, depriving centres of relevant and 
regionally grounded strategic advice. 


At midterm, the South Asia centre is the only centre with a functioning RAC. Other regions are still 
in the process of establishing committees; the AA centre has identified all RAC members and the 
LA centre has identified some members. 


According to their project documents, the centres are obliged to seek CLEAR Board approval for 
the nomination of RAC members. In one centre this contributed to delays in confirming RAC 
composition, given that some initially proposed candidates were not endorsed by the Board. This 
put the centre into an awkward situation in relation to the rejected candidates. Other centres 
reported that budget constraints (AA centre) or the absence of agreed upon mechanisms for 
working together (EA centre) prevented the RAC from meeting since their appointment. 


The RAC in South Asia has met twice and the experience to date, while limited, supports the 
assumption that RACs can provide CLEAR centres with relevant and regionally grounded strategic 
advice. To prove this assumption, additional experiences gained in other contexts are required. 
The four centres without functional RACs have had fewer, or at least less systematic and 
transparent opportunities than the one in SA to elicit input and validation of their (planned) work 
from regional experts. 


61 One reason relates to financial constraints-the AFDC has the resources to support the training costs of 
government, but not CSO representatives. 
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3.1.3 Centres’ performance 


Finding 17: CLEAR objectives have been relevant in light of the interest in and commitment 
to managing for results shown by government and non-government actors in a 
wide range of countries in all five regions. 


In all five regions, overall CLEAR objectives as well as the specific activities conducted by the 
regional centres have been relevant in view of existing gaps in the supply of and demand for M&E 
services and related capacity building. There were, however, variations in the extent to which 
centre activities were focused on the most pressing needs in the respective context. In countries 
with low pre-existing M&E capacity and culture, some centre activities (e.g., the provision of impact 
evaluation courses in AA) were considered too advanced for the level of experience, knowledge, 
skills and demands of most regional stakeholders. 


To date, centre activities have been generally in line with centre strategies (implicit or explicit). 
However, given the absence of detailed strategies in some centres (FA, EA), and the absence of 
explicit theories of change in all centres, it is not yet evident if or how individual centre activities 
and/or client relationships fit into a larger and longer-term vision of ‘success’ in the respective 
context. 


Finding 18: All five centres have met most of the midterm targets defined in the CLEAR 
results framework, which focused on the selection and establishment of 
regional centres. To date there is limited evidence of their contributions to 
CLEAR’s envisaged higher level development results. 


As illustrated in CLEAR annual reports, all five centres have fully or mostly achieved the midterm 
targets that focused on the establishment of centres and their ability to provide a variety of capacity 
building services for M&E and RBM. 


At midterm, while there is evidence that the centres have reached a considerable number of 
individuals and organisations, there is limited and largely anecdotal information on whether this has 
also (or is likely to) influenced these actors and/or their organisations and with what effects. 
CLEAR centres are only starting to conduct regular follow up with clients and collect data in order 
to capture the longer-term effects of their work on individuals and organisations. Also, in the 
absence of region or country-specific theories of change it is not consistently evident how 
individual achievements are 
envisaged to eventually contribute 
to broader, system-level changes. Individual level 
Outlined below is an overview of Increased awareness (of what M&E/RBM are and their benefits; 
the types of centre achievements different types of evaluation; gaps in one’s own knowledge or skills) 
noted to date. We have mapped 


i ; Increased confidence/motivation to engage in M&E or RBM; 
these achievements against the 


four overall outcomes in the Increased knowledge/skills (e.g. related to different evaluation 
CLEAR Theory of Change.®? approaches and methodologies; how to manage evaluations) 
Strengthened capacity to Organisatonallevel 

produce and use evidence Availability of organisation-specific toolkits, guidelines, policies 
Almost all centre achievements Access to international experts (Impact Evaluation, Performance 


made to date relate to creating Budgeting) to provide inputs and advice 


62 Given the noted absence of clearly elaborated pathways of change and the fact that CLEAR reports still 
tend to be activity, rather than results focused, this process was based on i) how CLEAR centres themselves 
mapped achievements in the latest CLEAR annual report and ii) the evaluation team’s assessment of which 
outcome(s) an achievement was most relevant to. 
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favourable conditions that — in the longer term — have the potential to contribute to individual actors 
or organisations producing (and eventually using) more or better evidence. Examples are shown in 
the sidebar.% To date, there is only very limited and largely anecdotal information available on 
whether and how individuals or organisations have actually applied the knowledge and skills, 
policies and tools gained through CLEAR interventions, and with what effects. 


Expanded professional expertise in regions 


All centres conducted at least some interventions aimed at strengthening the professional capacity 
of actors (individuals and organisations) outside of the centres’ home countries, as well as efforts 
focusing on supporting the professionalization of M&E and RBM. 


Relevant types of achievements include the creation of more and/or better® opportunities for 
exchange and learning for evaluation professionals in the region, due to financial and technical 
contributions to professional networks at national and regional level, and at CLEAR-organised 
events such as the global forum or regional roundtable discussions. The evaluation did not find, 
and it may be premature to expect, evidence of positive changes deriving from these interventions. 


Enabling Environment and Demand for M&E 


The CLEAR centres have not yet made explicit what aspects of the enabling environment (or types 
of demand) they consider possible or likely to influence, how, and with what envisaged results. 
Nevertheless, there is evidence of CLEAR centre activities have contributed to a number of 
positive changes in different types of demand, as outlined below. 


Demand for what? Indicators of changes in demand to date due to CLEAR interventions 


Demand for M&E Capacity Interest in and use of available (i.e. new) CLEAR capacity building 
Building services. 
Targeted client requests for tailored capacity building services through a 
CLEAR centre, e.g. following their participation in more generic offers (e.g. 
training events) from the respective centre. 


Demand for more or better Requests to CLEAR to: 


evaluations to be conducted; e Help strengthen M&E systems (at organizational, sub-national, or 
better M&E systems to be national level) 
established & functioning. e Provide assistance to plan, manage, conduct (meta) evaluations 
e Help develop or strengthen guidelines, frameworks to guide M&E 
work 
Demand for evidence No evidence at midterm. 


generated through evaluations 


$3 Some of the noted types of achievements correspond to the ‘intermediate capacity development 
outcomes” as per CLEAR’s document “Building Blocks of CLEAR’s Capacity Development Strategy’ (2013), 
which names the following such outcomes: Raised awareness, enhanced skills/knowledge; improved 
consensus/teamwork; enhanced networks; strengthened coalitions; new implementation know-how. 


64 In terms of the diversity and quality of contents addressed at, for example, Conclaves. 
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Using the framework suggested by Mackay,® most of the strategies employed by CLEAR centres 
to date that are primarily geared at influencing one or more types of demand (as noted above) fall 
in the category of ‘sermons’ (see sidebar). While results in awareness raising are challenging to 
measure, consulted CLEAR staff, 

clients and collaborators in all 


regions consistently noted that How to strengthen demand 

CLEAR had made important Mackay (2007) argues that demand for M&E can be strengthened 
contributions to raising the visibility through the following types of influences: 

of M&E among targeted Sermons - high-level statements of endorsement and advocacy 
stakeholders, and to increasing concerning the importance of M&E. These also include efforts to 
their understanding of the benefits raise awareness of M&E and to explain to government officials (or 
that the improved use of M&E other actors) “what's in it for them”. 

could have for achieving the goals Sticks — prods or penalties for organisations or individuals who fail 
of their unit or organisation, and in to take performance and M&E seriously. Common tools that act as 
identifying gaps in their ‘sticks’ include legal and policy frameworks and related guidelines. 
organisational, sectoral, or Carrots — ways of providing positive encouragement and rewards 
(sub)national systems. for conducting M&E and utilising the findings. These include, for 


example, public recognition or financial incentives to individuals or 


In comparison, the centres have organisations. 


done relatively less in relation to 
“sticks”.© In some cases, interventions were not always deemed necessary; for example in many 
Latin American countries, frameworks were already considered fairly strong, and more focus was 
needed to implement them. In other cases, interventions were not feasible at a particular time; for 
example in India, the CLEAR centre has been awaiting the outcome of the elections and 
implications for the sustainability and positioning of the Independent Evaluation Office (IEO) before 
deciding whether and how CLEAR will support the IEO in pursuing changes in the existing legal 
provisions for M&E. 


All CLEAR centres have made efforts and contributions to creating or strengthening “carrots”. 
Relevant in this regard are, for example, centre efforts in bringing together different actors (in 
heterogeneous or homogenous groups) to discuss, explore, and learn about M&E and RBM, 
thereby fostering the potential for synergies among these actors. 


Innovation in M&E 


All of the centres have played a role, albeit to varying degrees, in introducing their clients 
(individuals and organisations) to new M&E-related ideas, tools, or approaches. Related 
achievements have the potential to contribute to innovations in M&E in the longer term,” but, in 
our view, do not yet constitute innovation. 


65Keith Mackay (2007): How to Build M&E Systems to Support Better Government. World Bank/IEG. 
Retrieved on May 5 from: http://www.worldbank.org/ieg/ecd/docs/How to build ME gov.pdf 


66 Examples of related achievements include CLEAR EA support to the Government of China for issuing new 
regulations on performance evaluation on IFI projects; CLEAR AA contributions to a review of four DPME 
guidelines on different types of evaluations, which were included in the national evaluation plan for South 
Africa; and the LA centre’s work with the Peruvian ministry of Women and Vulnerable Populations around 
creating a monitoring index based on administrative and census information. 


87 As noted in the Glossary and section 2.2, the evaluation considered a product, process, service or a 
technology to be an innovation in M&E if at least two of the following criteria were met: a) There is significant 
process improvement, i.e. the innovation notably influences how M&E is done, b) There is catalytic change; 
i.e. an innovation is not just a cheaper or faster way of doing the same thing as before, c) The innovation is 
concrete, i.e. while an idea of theory can lead to innovations in how M&E is conducted, it does not itself 
constitute an innovation unless it is applied. (See: UNDP, 2013). 
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This is neither surprising nor concerning at midterm. However, the evaluation noted a lack of clarity 
and shared understanding among CLEAR centres of what counted as “innovation in M&E”, making 
it more difficult for the centres to systematically strive towards this objective. The notion of 
innovation is not prominent in any of the centres’ deliberations on their respective strategies or 
envisaged results Furthermore, none of the current indicators in the CLEAR results framework 
require the collection of data suited to provide an indication of whether and how centres have 
contributed to innovation, and with what benefits. This limits the initiative’s ability to consistently 
collect data in this regard. 


3.1.4 Efficiency and viability 


Finding 19: Due to their location in host institutions, centres had to adhere to a variety of 
different rules, regulations and requirements that contributed to some 
inefficiency. 


Several centres experienced delays and challenges due to difficulties in aligning host institution 
rules and procedures with World Bank requirements. In several cases (especially in AA and FA) 
this led not only to delays in the transfer of funds to the centres, but also to inefficient use of 
professional staff, given that they were tied up with administrative tasks. 


On the positive side, all CLEAR centres have been able to leverage some additional resources for 
operation and activities. In the LA and EA centres, the leveraged resources have been 
considerable and have surpassed the grant amounts that the centres received from CLEAR. 


Finding 20: The likelihood that Regional Centres and their services will continue without 
CLEAR funding varies — from low in Anglophone Africa to very strong in Latin 
America and South Asia. 


A review of the likely longer term viability of the Regional Centres indicates that their prospects 
vary. Variations are due to differences in the extent to which the centres have already engaged in 
(and have the capacity to engage in) planning for financial sustainability; their current integration 
into and types of support received from their host institutions; the extent to which the demand for 
the types of services they provide continues or increases in their national and regional contexts; 
and the number of actors willing and able to pay for these services. 


Viability is, however, not just a question of whether a particular centre canbe sustained, but also 
whether the likely benefits and incentives linked to its continuation are sufficiently strong that the 
host institution (and centre staff) is inclined to pursue this goal. This was raised as a possible 
concern in East Asia, given the small amount of CLEAR grant money made available to the EA 
centre. 
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3.2 CLEAR Anglophone Africa (AA) Centre 
3.2.1 AA Centre Profile 


Selection and establishment 


The University of Witwatersrand, 

South Africa (Wits) was selected The CLEAR centre is located within the Wits School of Governance, 
from 24 applicants from six formerly known as the Graduate School of Public and Development 
countries in Southern Africa to Management (P&DM). Besides CLEAR, the Wits School hosts two 
serve as the CLEAR regional centre other centres — the Centre for Defence and Security Management, 


à and the Centre for Public Enterprises © 
in October 2010. The centre 
became operational in June 2011.° See sidebar. 


As noted in its proposal to host CLEAR, Wits is partnering with the Ghana Institute of Management 
and Public Administration (GIMPA), and the Kenya School of Government (KSG) to serve as 
CLEAR affiliate centres and 

facilitate working across national 


borders in the region. AA Centre Institutional Funding Sources 


Financial resources World Bank 


A first grant agreement between the | Food and Agriculture Organization (FAO) 
World Bank and Wits was signed In 
December 2011, for a grant of 

USD 940,513 for the period June Japan International Cooperation Agency (JICA) 
2011- January 2013. A second German Technical Cooperation Agency (GTZ) 
grant agreement for USD 2,997, 
325 was signed in May 2014 for the 
period 2014-2018; at the time of Rockefeller Foundation 

writing, World Bank records indicate 

that approximately USD 200,000 has been disbursed, which the AA centre reports has been 
largely used to retrospectively finance previous activities (including its attendance at the 2013 
Global Forum and the 2014 AfrEA conference). Total projected funds for the AA centre up to the 
end of FY 2018 (including grants, selection process, and Secretariat support) amount to 

USD 5,073,687.” 


In addition to the World Bank administered CLEAR grant, the AA centre has generated revenue 
from a number of bilateral and multilateral donors, national governments and foundations since its 
inception for research, training or consultancy services as well as revenue earned from individual 
course participants. For example, between 2012 and December 2013, the AA centre had 
generated USD 933,000 from other sources of funding, as compared to USD 940,000 from the 
original CLEAR grant. 


Staffing 


The centre’s first staff members, an acting Director”? and an administrative officer, commenced 
their responsibilities in February 2012 (8 months after the centre’s establishment). By August 2012, 
its staff grew to include a technical expert, a programme administrative officer, and a part-time 


UK Department for International Development (DFID) 


Government of South Africa 


$8 Source: http://www.wsg.wits.ac.za/centres 


69 While the grant was signed by both parties in February 2012, retroactive financing allowed the centre to 
claim expenses backdating to May 2011. 


70 Constituting the highest grant amount among the currently active CLEAR centres. 


71 The acting Director was confirmed as Director in July 2013. 
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bookkeeper. At the time of writing (September 2014), the AA centre staff includes an Acting 
Director (the former Director resigned in February 2014), a programme administrative officer, and a 
programme management officer responsible for financial management. Since September 2014, the 
centre has contracted a programme support officer on a shared basis with the Wits School of 
Governance. Two CLEAR associates are further contracted for particular tasks. There are two 
CLEAR coordinators in each of its affiliates in Kenya and Ghana. 





Centre activities Type of activity February 2012 
Until the end of 2013, the AA centre had eats 2013 
implemented approximately 60 capacity 
building interventions, as shown in the Training 27 
sidebar. 
Centre clients have been predominantly Knowledge exchange seminars 10 
representatives from government units Ree enor a 
(60%); others clients have included ON nee 
individuals from non-government Collaboration with M&E networks 6 
organisations, academia, post-graduate 
students and M&E professional networks. Diagnostic 1 
There have been relatively few clients from 
the private sector or philanthropic Knowledge resource 5 
organisations. 

Evaluations, assessment, advisory 4 
3.2.2 AA Centre Context 

Advocacy, promotion, and other 2 


Finding 21: At the time the AA centre 
was established, the demand for evidence in decision making for improved 
development results was relatively modest in Anglophone Africa. By 2014, 
there is evidence of increased awareness and interest in the value of 
performance information, but demand is isolated and required most often for 
accountability rather than learning or decision-making purposes. 


Prior to the establishment of the AA centre, various stakeholders carried out some analysis of the 
use of evidence (generated through evaluations or performance management) in decision making 
in Anglophone Africa, as well as reviews of the demand for and supply of training and other 
support services intended to build national and regional capacities in evaluation and performance 
management. Key characteristics of the regional context at the time the AA centre was 
established”? include the following: 


e Historically weak enabling environments with low effective demand for M&E services 
and for evidence deriving from M&E due to lack of incentive structures, such as policies 
and frameworks. However, there was one notable exception, South Africa, which was in 
the process of putting stronger incentive frameworks in place. 


e Limited awareness and understanding among potential users, especially within 
government institutions, of the benefits inherent in evidence generated through M&E. 
This was, for example, reflected in the absence of M&E-related content in public 
management degrees. As a result, “doing M&E” was often equated with conducting 
expenditure or output level monitoring activities. Overall, infrastructure support for and 
expertise in evaluation were considerably weaker than for monitoring. 


72 As described in the AA centre project documents, and confirmed by literature review and stakeholder 
consultations. 
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e Limited supply of high quality M&E services, with many practitioners lacking knowledge, 
skills and experience in core competencies for scientific data collection and analysis. At 
the time, there was one national Voluntary Association of Professional Evaluators 
(VOPEs) in South Africa and one for all of Africa, AfrEA. 


e A growing, but scattered and not consistently competent regional supply of M&E 
expertise in the region, resulting in a continued reliance on expertise from Europe and 
North America. 


e Agrowing momentum among M&E promoters and practitioners to adapt existing M&E 
tools and practices (or develop new ones) in order to meet the specific needs of the 
regional, national, and organisational contexts in Africa. 


While the types of opportunities, needs, and challenges were similar across the region, they 
manifested differently in each country.” 


Interviews carried out in February 2014 with the AA centre stakeholders in the region as well as a 
review of the demand and supply studies carried out by the AA centre in 2013, indicate that there 
has been notable growth in interest in the demand for and supply of M&E skills training in the 
region. Since 2011, several universities and colleges in AA (including Africa Nazerene in Kenya; 
Uganda Technology and Management and Makerere in Uganda; and Capetown, Pretoria and Wits 
in South Africa) have launched certificate, diploma and/or degree programmes in evaluation that 
have steady and increasing enrolment; several national governments in the region (e.g. Zambia 
and Ghana) have taken (or are in the process of taking) steps to underline the importance of (and 
build their own government’s capacities) in generating information about and reporting on their 
government's performance, or engaging in capacity building activities to enhance the use of 
evidence in decision making by African government cabinets;”° and a couple of VOPEs in Ghana 
(Ghana M&E Forum) and Kenya (Evaluation Society of Kenya) are becoming stronger. 


On the other hand, interviews and document reviews suggest that the overall context related to the 
use of evidence in decision making has not significantly changed since the establishment of the AA 
centre. Persons interviewed indicated that M&E systems and capacities are still being developed in 
many countries, and when they exist, the information generated is still used mainly for 
accountability purposes. More time and institutional “buy-in” will be required before information is 
used to inform decision making. Even in a country like South Africa, which is considered to be 
among the countries with a more mature understanding and capacities, some interviewed persons 
identified instances of the government’s tardiness in communicating the results of certain 
evaluations that did not rate government programmes positively (particularly at election time), for 
fear of political reprisal. However, the limited use of evidence (including evaluations) in decision 
making is not unique to Anglophone Africa; it is known to happen in places with considerably more 
mature systems (including North America and Europe). 





73 Please see ANNEX M for further details on the national contexts of South Africa, Kenya and Ghana. 
74 Stellenbosch University, South Africa established its post graduate diploma in M&E prior to 2011 


75 Through a three year Africa Cabinet Decision-making programme launched in 2014, DfID is providing 
support to the Africa Cabinet Government Network to help build the capacity of up to 12 African 
Governments to make better evidence-based collective Cabinet decisions. This will lead to more 
systematic and better implemented policy, contributing to national development goals, poverty 
reduction and improved quality of life for citizens. (http://www.cabinetgovernment.net/) 
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3.2.3 AA Centre Strategy and Capacity 
Strategy 


Finding 22: The AA centre’s programming strategy has evolved from being responsive to 
one that is considerably more focused and contextually realistic. The 2013 draft 
strategy has not yet been fully applied due to long delays in being able to 
finalize the procedures necessary to receive the CLEAR grant as well as 
current gaps in AA centre leadership and capacities. 


An external review of the AA centre in 2013 reported that the centre lacked a clear and well- 
articulated strategy for moving from vision to action.” A review of AA centre documents and 
interviews with key stakeholders suggest that this was the consequence of two factors: its 
programming approach in the first two years which emphasised the delivery of training, and the 
absence of coherent strategic directions and guidance from the CLEAR Board, the Secretariat and 
the host institution Wits, as explained below. 


In the first couple of years, much of the centre’s programming centred on the delivery of various 
capacity development interventions for individuals and organisations in the region including RBM 
and performance-based budgeting courses with the Government of South Africa, and open 
enrolment impact evaluation courses. Many of the open enrolment activities were conducted with 
the purpose of making the centre known among a variety of (potential) client organisations in the 
region. Over time, the centre started to provide tailored and more comprehensive capacity building 
services to individual clients in response to their requests (with a large focus on the South African 
government) providing training as well as advisory services and technical assistance. The AA 
centre also responded to other opportunities as they presented themselves, including carrying out 
research about the supply and demand for M&E in certain countries in the region (funded by 
DFID), developing customised courses for the Food and Agriculture Organization (FAO), and 
offering sessions at various 

conferences (including SAMEA and 


AfrEA conferences). AA Centre Priorities — Strategic Questions 


At the same time, the AA centre Target groups: To what extent should civil society be a priority? 
(along with other CLEAR centres) 
struggled to clarify where its 


strategic directions should come ee | 
from. In the absence of a Regional Quality: How is quality defined in a low capacity environment such 
Advisory Committee and modest as Africa (meeting international standards or something else?) 


guidance from the Wits School of Centre viability: To what extent should the centre focus on 
Governance, the centre sought generating new resources vs. utilising the CLEAR grant? 

and/or received advice from several Proactive vs. responsive: To what extent should the centre play a 
other sources including the CLEAR proactive role in the evaluation field on the continent vs. responding 
Board, individual Board members, to clients with consulting services? 

the Secretariat, and individual M&E 

experts. However well-intentioned, the advice was and still is sometimes contradictory, leaving 
centre staff with questions on some strategic directions. See sidebar. 


Centre research: Should it be relevant to university and/or CLEAR 
priorities? 


The AA centre developed a draft strategy (project document) in December 201377 that was 
subsequently approved by the CLEAR Board. However, as of September 2014, the strategy 
document had not yet been implemented due to a combination of factors since March 2014 


76 Mindfarm organisational assessment, August 2013 
77 CLEAR-AA Project Document .December 2013 
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including the departure of the centre Director, limited AA staff capacities, and continued financial 
constraints related to long delays in its receipt of World Bank funds (see also finding 30). 


The strategy is a thoughtful document that has the potential to provide a solid basis for the future 
work of the centre for several reasons: 


e The document provides an analysis and synthesis of the specific challenges and needs 
in the AA regional context, and aligns them with the four overall CLEAR outcomes. 


e The strategy provides insights into the centre-specific Theory of Change” i.e. it 
identifies concrete, region-specific changes that the centre envisages to contribute to. 
Specifically, it indicates the intent to focus centre activities on supporting efforts to 
strengthen capacity to implement country-led evaluations within government. This is 
done in view of the broader goal to contribute to governments using evidence to make 
effective decisions and to improve development outcomes. 


e The document describes the key types of services the centre intends to offer, and 
indicates its continued commitment to grounding services in detailed diagnostics of the 
specific needs of the respective national or organisational contexts. Besides helping to 
develop the capacity of national M&E units, the strategy also notes the centre’s intent to 
work with relevant civil society organisations, and to contribute to development of 
templates and tools and region-specific M&E literature. 


e The strategy comments on how the centre interprets the notion of being a ‘regional’ 
centre, noting the intent to focus on the national priorities of four countries,’? which were 
identified based on a transparent assessment and rating system. Broader regional 
engagement is envisaged to identify innovative practices emanating from different 
sectors and different actors from across the region. 


e Finally, the document describes the roles that different types of partnerships are 
envisaged to play and how they will contribute to results. Similarly, it outlines criteria for 
prioritising demand for services from different government units. 


Overall, the analysis contained within the strategy is helpful in view of clarifying the AA centre’s 
choices based on its assessment of what constitutes ‘strategic’ choices. One potential limitation in 
the usefulness of the draft strategy is that the only measures of success are the indicators in the 
overall CLEAR results framework. While this is consistent with CLEAR reporting obligations, it 
does not clarify what “success” in the Anglophone Africa region would look like. 


78 While the CLEAR centres were, in theory, free to develop their own Theories of Change, in case of the AA 
centre the Secretariat suggested using the overall CLEAR program ToC in order to complete the already 
delayed Project Document, and due to the fact that, at the time, the Centre had not yet envisioned a centre- 
specific ToC. 


79 South Africa, Zambia, Kenya, and Ghana 
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Human resources 


Finding 23: The AA centre’s human resource capacities have not been well aligned with the 
demand for its M&E capacity building services or with the administrative 
requirements of establishing and running the centre. 


Since its establishment, the AA 


centre has experienced human Status of Mindfarm Organisational Assessment (2013) 
resource challenges. An Recommendations 

organisational assessment of the Establish the RAC: Done, although it did not meet due to budget 
AA centre in 2013 (conducted at the | considerations 

request of the CLEAR Board and 


with agreement by Wits that the Address outstanding World Bank administrative requirements: Done 


study would be useful), identified Appoint a senior finance person for the Centre: Done (February 
significant gaps in the centre’s 2014) 
managerial and professional Develop a staffing plan: Done 


capacities and recommended how 
to strengthen centre capacity. The 
recommendations had been partly 
acted upon at the time of the 
evaluation; the remainder may be Build capacity in Wits: Not done, due to budgetary constraints 
addressed Onge a new centre Develop a communications strategy: Started, but not completed due 
director is appointed and resources to budgetary constraints 
are available. Evaluation findings z ne : ; 
confirm and support the evelop a sustainability strategy: Not done due to capacity 

$ ‘ constraints. 
observations and recommendations 
of this assessment. Key issues are 
noted below. 


Hire a senior-level M&E advisor: Not done, due to budget 
constraints after June 2013 


Determine what success looks like for AA Centre: Not done 


Number of staff and transition: The centre has been almost exclusively reliant on an Acting Director 
since its inception (with the exception of eight months from July 2013 to late February 2014. The 
centre has experienced staff turnover in all positions since it was established,®°partly due to the 
relatively tenuous viability of the centre in the university (see finding 31) and related staff insecurity. 
At September 2014, the AA centre was still in the process of recruiting a Centre Director. 


Managerial capacity: Since becoming operational, the centre lacked staff experienced and able to 
efficiently navigate and satisfy both World Bank’s and Wits’ rules and regulations. As a 
consequence, the centre experienced challenges and delays in satisfying administrative 
requirements of both institutions, which contributed to multiple inefficiencies (including delays in 
approvals, funding, staff appointments and so on with adverse effects on the centre’s capacity to 
carry out programming activities. Over time, the CLEAR Secretariat has provided support to the AA 
centre to assist it with some day-to-day operations. 


Professional capacity: In terms of experienced professional staff, the centre relies on the Acting 
Director, a couple of Wits researchers, and several international and regional associates who are 
contracted on an ‘as needed’ basis Interviews with all centre stakeholders (including clients) 
pointed to significant gaps between the number and qualifications of centre staff and the needs 
and demands of clients. The over-riding view was that the centre relied exclusively on the former 
Centre Director, who was highly respected among CLEAR clients and partners for the quality of his 
support and flexibility in tailoring support to clients’ needs. However, the reliance on one individual 
was also repeatedly raised as a concern about the depth and scope of the centre’s professional 


80 Including the Centre Director, a couple of programme management officers, a couple of technical 
specialists 
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capacity and expertise. Staff recruitment was hindered by a combination of factors, including the 
need to respect the hiring policies of Wits, the need to satisfy the terms and conditions of the 
agreement between the World Bank and the university, the tenuous nature of the centre itself (a 
new entity in the university with ongoing cash flow challenges which inhibited its ability to recruit 
staff and associates), as well as a reported shortage of experienced (and particularly African) 
professionals with the necessary expertise to fill the centre’s staffing needs. The centre instead 
relied on a pool of associates to deliver training in a number of countries as required. 


Centre capacity building: AA centre staff and some partners have undergone on the job training 
and have attended capacity building events such as the IPDET course and IDEAS conferences. 
However, the centre has not yet developed a tailored plan that identifies required staff capabilities 
and how to acquire them. This is understandable, given staffing challenges identified above. 


Inter-institutional linkages 


Finding 24: While the AA centre has established a couple of partnerships with like-minded 
organisations for mutual benefits, particularly in South Africa, it lacks sufficient 
partnerships to realise its vision and objectives. 


Like all CLEAR centres, the AA centre has a broad mandate and vision, but relatively modest 
financial and human resources to realise its objectives. One strategy to address this challenge is 
collaboration with other national and regional partners for mutual benefits. 


A review of the AA centre’s work from 2011-14 indicates that it has been particularly successful in 
building such partnerships with the South African government, and particularly the Department of 
Performance Monitoring and Evaluation (DPME) housed in the office of the Presidency. Interviews 
and document reviews provided evidence of a strong, mutually beneficial and respectful 
relationship between the AA centre and DPME that builds on some strong linkages between Wits 
and the RSA government that existed before the launch of the CLEAR initiative. While the AA 
centre services have supported DPME in building its capacities in several ways (see finding 28), 
the two organisations have collaborated on several initiatives including drafting parts of the South 
African National Evaluation Policy Framework, developing standards and competencies for 
implementing evaluation in South Africa. The AA centre’s relationship with DPME has also helped 
to positively influence the centre’s visibility and reputation, as well as its access to other influential 
actors in South Africa and beyond. 


Over the review period, the AA centre also worked closely with DFID on topics of mutual interest, 
including studies on the demand for and supply of evaluation in Africa.8' The centre has also 
collaborated with the African Evaluation Association (AfrEA) in several ways, including: 


e Attending/participating in discussions with AfrEA and others, including the South African 
Monitoring and Evaluation Association (SAMEA), in 2012 to identify potential areas for 
collaboration 


e Launching a forum with AfrEA entitled "Thought Leaders in Development and 
Evaluation”& which took place at the Bellagio Centre with the support of the Rockefeller 
Foundation in 2012 


e Facilitating a Made in Africa evaluation stream at AfrEA’s 7th biennial Conference in 
Cameroon in March 2014. 


81 http:/Awww.clear-aa.co.za/wp-content/uploads/201 3/09/20 1402-Demand-and-Supply-Final-Report. pdf 


82 This initiative was intended to give impetus to efforts to develop original and influential African 
contributions that can strengthen evaluation theory and practice on the continent and worldwide, and help 
position the AfrEA Made in African Approach to Evaluation Source: http://www. afrea.org/?q=node/3 
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At the same time, the AA centre did not have any effective partnership agreements with other 
institutions or organisations to assist it in delivering technical services in different parts of the 
region. This includes its relationships with KSG in Kenya and GIMPA in Ghana, which were in 
relatively early stages of developing their own institutional capacities, and thus had limited capacity 
to assist the AA centre. Moreover, the ‘fee for service’ agreements between the AA centre and 
these affiliates were difficult for the affiliates to accommodate due to their own cash flow 
constraints.% Finally, the agreements do not reflect a jointly developed and agreed upon longer 
term vision and strategy. 


The combination of human resource capacity constraints and its limited number of working 
partnerships have limited the centre’s potential in responding to client demand and in playing a 
more proactive role in Anglophone Africa. This is further evidenced by the limited number of AA 
centre activities that have taken place since March 2014. 


Host institution arrangement and governance 


Finding 25: The host institution arrangement between the AA centre and Wits has 
considerable potential for benefits for the university and the AA centre alike. To 
date, most of these benefits have not been realised. 


While Wits is only one of several universities in South Africa that is building its capacities and 
credibility in the M&E field, its long standing historical association with the African National 
Congress government meant that it had the necessary strong trust and relationships to engage 
meaningfully with the government on good governance matters (including encouraging the use of 
M&E in decision making), which provided the AA centre with several very valuable entry points. 
Second, the placement of the AA centre within Wits’ School of Governance, coupled with the 
university's research and education priorities and desire to be a pan-African institution, were 
congruent with the AA centre’s objectives. In addition, at the time that Wits was developing the AA 
centre proposal, some hoped that the AA centre would help to harness and secure pockets of 
interest in evaluation in other Wits’ schools (e.g. the Education and Psychology schools). These 
reasons made (and continue to make) the university a promising host for the CLEAR centre. 


To date, the most significant and 
broadly acknowledged benefit for 


the university emerging from Wits School of Governance/CLEAR Post Graduate diploma in 
hosting the AA centre was the M&E 

launch of the Graduate Masters Wits reports that the post graduate diploma is one of the most 
Diploma in M&E (see sidebar). On important outputs to date, noting that it is now the ‘most popular 
the other hand, interviews indicate diploma in the school and the country’, ‘fully institutionalised in the 
that the AA centre has not yet university and fully self-sustaining. 


realised its potential in furthering 
the university's research agenda, an important university priority. 


There are several reasons for these lost opportunities. The first is the AA centre’s human resource 
constraints, noted above, which have handicapped its potential. The second is related to some 
shortcomings in the support provided to the AA centre by Wits. On the one hand, some influential 
individuals in Wits’ senior leadership supported the notion of CLEAR and saw the establishment of 
a CLEAR centre as a positive and potentially valuable asset. On the other hand, as of February 
2014, the AA centre had not yet secured the attention and support from the various schools and 
Deans within Wits that was needed to support its institutionalisation. This was illustrated, for 


83 The terms of the affiliates’ contracts with CLEAR AA meant that they were reimbursed long after incurring 
expenses for activities they carried out on behalf of CLEAR AA. The multiple approval processes associated 
with processing each invoice led to very long delays in reimbursing the affiliates, causing them cash flow 
difficulties. 
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example, by: the continued absence of a university-wide Board that was supposed to be 
established to guide the centre; the ad hoc involvement of other schools in Wits that were 
supposed to be involved in supporting the work of CLEAR (due in part to the retirement of a couple 
of academics in the schools of Education and Psychology whose personal interests in evaluation 
were not pursued by their successors); and the lack of involvement of the Dean of Research in 
CLEAR activities. 


In addition, as noted in the 2013 capacity assessment study, Wits’ infrastructure and processes 
repeatedly made it difficult for the AA centre to do business in a timely manner, and have not been 
conducive for a centre like CLEAR that would benefit from cross-cutting partnerships and 
innovative business arrangements. 


Finding 26: The AA centre’s Regional Advisory Committee (RAC) has not yet met formally 
or provided input to the centre’s work as a group. This has limited the centre’s 
ability to tap the strategic guidance that this body might offer. 


The AA centre has identified four members of its Regional Advisory Committee, including one 
individual each from Ghana, South Africa and Kenya, as well as one representative from UNDP.84 
The RAC membership was approved by the CLEAR Board in July 2013, i.e. about two years after 
the centre became operational. 


The delay in establishing a RAC was due to a number of reasons. The initial Terms of Reference 
for the AA centre RAC were not approved until May 2012. The process of establishing a RAC was 
then put on hold due to the uncertainty around the AA centre Director’s position. In February 2013, 
RAC members were proposed, but not all of them were approved by the CLEAR Board. To the 
best of our knowledge, RAC membership in AA has not been resolved. 


The AA Centre Director sought feedback from individual RAC members when drafting the centre 
strategy in 2013, but at the time of the evaluation the committee had not yet held a formal meeting 
due to insufficient financial resources. This represents another lost opportunity for the centre to tap 
into the knowledge, experience, contacts and advice that its RAC can offer to its future growth and 
development. 


3.2.4 AA Centre Performance 
Relevance 


Finding 27: The AA centre objectives and activities have been relevant in view of the 
identified gaps in the existing supply of and demand for high quality M&E 
services in the region. 


The draft CLEAR AA strategy (2013) notes that governments in Anglophone Africa frequently lack 
evidence to respond to citizens’ increased demand for accountability, and typically do not engage 
in monitoring and evaluation as means to support effective responses to those demands. The draft 
strategy identified four reasons for what it labelled a lack of a culture of learning for improvement 
as follows: 


e Lack of enabling environment for demand: There is no shared body of knowledge on 
M&E across the continent, despite the fact that supranational bodies like the African 
Development Bank are expressing an interest in M&E. 


e Limited capacity to use evidence: Incentive structures such as policies and 
frameworks have only now started to be formulated on M&E but they lack basic 
frameworks such as widely accepted definitions of M&E core competencies. 


84 Project Document (2013), p.28. 
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e Limited supply of M&E professionals: Few management experts have adequate 
knowledge on M&E (M&E is taught as an ‘additional’ competency separate from other 
management skills). 


e Limited innovation in the region of M&E practice: Innovations are generally not 
being undertaken. Civil society organisations are undertaking more evidence-based 
critiques of government's role but these challenges are not being addressed in internal 
learning and accounting processes. 


Given this assessment of the context, AA centre activities to date have been relevant in that they 
have generally been aligned with the types of challenges noted above, as well observed regional 
trends such as the growing demand for more region-specific tools and approaches to M&E. The 
centre has engaged in activities intended to increase awareness of existing gaps and shortcomings 
through individual (organisational) diagnostics, and assessment studies of three countries in the 
region, as well as the Made in Africa 


movement. 

In addition, the AA centre has Survey of Participants in Impact Evaluation Training in 
worked with M&E practitioners and | Anglophone Africa® 

managers to strengthen their All surveyed participants who attended impact evaluation courses 
knowledge and skills to indicated that the training was relevant to their work responsibilities. 
commission, manage, or carry out A smaller percentage (58%) reported that the training was relevant 
evaluations/generate evidence. In given their employer's policies, priorities and/or resources allocated 


annual reports, CLEAR AA reported for evaluation; one-third reported it was somewhat relevant. 


that its clients rated the overall 

usefulness of its activities at 4.46/5, and that 82% rated the usefulness (application of the courses) 
as good or excellent. A survey of individuals who participated in AA impact evaluation courses 
reported that the training they received was relevant to their work responsibilities (see sidebar), 
although they had various views on the relevance of the training given their employers’ policies and 
resources for M&E. 


Effectiveness 


Finding 28: The AA centre’s greatest contributions to date have been with a couple of key 
institutions in South Africa; these accomplishments provide a glimpse of its 
potential to make contributions elsewhere under the right conditions. 


Over the review period, the AA centre engaged in a large number of activities as summarised in 
section 3.2.1 and met or exceeded most planned outputs. A review of its performance over the 
review period indicates that it made several important contributions. 


Tailored organisational capacity development in RSA: The AA centre is one of several 
institutions in the Republic of South Africa (RSA) helping the DPME deliver training in M&E to 
local/provincial government staff. DPME is the client that the AA centre has perhaps worked with 
most, in terms of the number of activities and resource allocations (time and/or money). The AA 
centre has provided various types 
of support to the DPME which has 
included working alongside it in the In RSA, there are universities and private firms that provide some 
development and delivery of five in- | training and services similar to CLEAR, and even to the same __ 
service training courses that clients that CLEAR has, such as DPME. In this context, CLEAR is 
ener a not seen to be providing ‘unique services’ but rather is appreciated 
covered topics including how to for its holistic, flexible, response to client needs and because it is 
manage evaluations, how to seen to be driven by objectives other than profit. 


85 The evaluation team surveyed participants in IE courses facilitated by the AA centre to validate claims 
made by interviewed stakeholders about the utility of IE courses to some participants. 
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prepare management responses to evaluation, how to communicate evaluation results, how to 
develop Theories of Change and logframes. The centre also assisted DPME in developing 
evaluation design clinics for staff. See also sidebar. 


Interviewed DPME staff claim that their knowledge and skills have increased in the areas covered 
by in-service training supported by the AA centre. Further, they appreciated the connections and 
exchanges that they have had with the aid of CLEAR support for a study tour to CONEVAL, the 
Centre of Excellence in Evaluation in Canada, and to other countries (Colombia, Mexico and the 
United States) that are also striving to develop government evaluation capacities and performance. 
CLEAR also reviewed four DPME guidelines on different types of evaluations, which were later 
included in the national evaluation 

plan for South Africa. 


The AA centre also collaborated The South-South Roundtable engaged government and 
with DPME and others in the parliamentary leaders from eight developing countries (Benin, 
inaugural meeting of the South- Colombia, Ghana, India, Mexico, Peru, South Africa, and Uganda) 
South R d Table Di s ; on the basis of their practical engagement in the area of policy 
ih pores in 50 3 Sn development and evidence-generation and use. 

„l 


support southern decision makers With support from the Bill & Melinda Gates Foundation, the 
EAE ER from selected roundtable held its inaugural meeting from 11-13 November 2013 in 
Pretoria, South Africa. It was co-organised by CLEAR Latin 


developing countries to share America, the National Council for the Evaluation of Social 
experiences in the use of evidence Development Policy of Mexico (CONEVAL), the DPME, and the 
for measuring and improving the Policy Coordination and Delivery Unit of the Presidency of the 
effectiveness of policies and Republic of Ghana, with the collaboration of the CLEAR Center for 
programs (see sidebar). Anglophone Africa. 


Source: Report on South-South Round Table Using Evidence for 


Wits Master’s programme: The Better Policy making and Practice South Africa 2013 3° 


AA centre contributed to the 
creation of a diploma and a 
Master’s programme in M&E at Wits. The diploma programme had 40 students in its first cohort in 
2013, selected from 400 applicants. In 2014, it had 180 graduate students registered, an 
impressive increase after a couple of years of operation. In addition, the university now also has 
several Master’s degree students and PhD students, from a baseline of zero in 2012. 


M&E in Kenya: In Kenya, a two-week CLEAR training contributed to the Government of Kenya 
initiating further collaboration with CLEAR to roll out its M&E system in 47 countries. 


Impact evaluation training: The AA centre has received largely positive feedback from 
participants in the Impact Evaluation courses it has offered. The majority of respondents to a 
follow-up survey carried out as part of this evaluation?” indicated that they had been able to utilise 
the knowledge acquired from the training; one-third reported extensive use, and the remainder 
reported some use. However, the interviewed course facilitators from Wits flagged their concerns 
about the limited relevance of this training, given the relatively modest capacities of most African 
governments in evaluation and other more basic and pressing priorities. 


Innovation: As noted earlier, the AA centre collaborated with AfrEA (with Rockefeller Foundation 
support) in 2012 on a regional forum to explore a Made in Africa approach to evaluation. The AA 
centre representatives indicate that these activities enabled actors from the development and 
evaluation sector to interact, supporting CLEAR's overall vision to encourage the use of evidence 
from M&E for decision making so as to enhance developmental results. The AA centre reports that 


86 http://www.clear-la.cide.edu/sites/default/files/SOUTH%20SOUTH%20ROUNDTABLE%20REPORT.pdf 


87 The AA centre obtains feedback from course participants at the end of training programmes only; it has no 
established process in place to monitor, track and report on the medium to longer-term effects of its training 
(or indeed other types of support) on participants or their employers. 
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these activities have informed the 2013 draft the AA centre strategy, as well as new work in Africa 
being initiated by AfrEA on evaluation competencies. Other consulted stakeholders, however, 
expressed some disappointment with follow-up to the Bellagio event, describing it as a lost 
opportunity for change. While the Made in Africa approach is clearly innovative, it will likely take 
time and considerable persistent engagement and support by key stakeholders (including CLEAR) 
for the approach to gather additional momentum and influence change. 


Exploratory case studies on African Monitoring and Evaluation Systems led by the AA centre and 
DPME using Benin, Burundi, Ghana, Kenya, Senegal, South Africa and Uganda as cases have 
become reference material for understanding the southern African context for M&E. Several 
consulted stakeholders saw the potential of the publication to help governments develop or 
strengthen M&E system on the continent. 


Promising strategies: The review of the effectiveness of the AA centre’s work in South Africa to 
date reveals several strategies which have the potential of being replicated with other partners in 
South Africa or elsewhere in the future. These include: 1) the provision of ongoing, professional, 
tailor-made technical support; 2) providing clients access to the AA centre’s valued networks and 
contacts with selected strategic actors (mainly DPME); 3) the adoption of cost-sharing practices 
and joint collaboration with DPME, which provide mutual benefits for DPME and the AA centre and 
demonstrate strong ownership of both entities. 


A final comment relates to the AA centre’s development partners over the period, which tended to 
be mainly government agencies. The centre generally engaged modestly with civil society and the 
private sector in AA countries other than providing training opportunities; better engagement with 
civil society is part of AA’s 2013 draft strategy. 


Finding 29: The AA centre has made several, albeit fragmented, efforts to engage with 
actors from other countries in the region and on regional issues. The centre’s 
revised strategy (2013) outlines a more focused and deliberate approach to 
regional engagement. 


Over the review period, the AA centre provided a number of training opportunities in countries 
other than South Africa, offered open enrolment training courses open to participants from other 
countries, and included 11 country diagnostic assessments since 2012. In addition, as noted 
above, it engaged in regional 


evaluation networking events, To date, the AA centre has conducted activities in South Africa, 
notably around the Made in Africa Ghana, Kenya, Uganda, Benin, Malawi, Rwanda, Ethiopia, Zambia 
approach. See sidebar. and Burundi. Training and workshops conducted in South Africa 

included participants from 22 other African countries, including some 
Nevertheless, the centre’s regional francophone countries.® 


work has been fragmented, without 

clearly formulated midterm or longer term strategic priorities and formulated results. Limited centre 
capacity (including in affiliate centres) has not permitted ongoing and significant presence in 
countries other than RSA. 


The AA centre’s strategy (2013) indicates that the centre’s future work will focus on four countries 
(South Africa, Zambia, Ethiopia and Ghana), and that in certain other countries (namely Kenya, 
Rwanda, Nigeria, Botswana, Uganda and Tanzania) it will limit its engagement to outreach and 
awareness raising activities and selected demand-driven interventions with individual client 
organisations. To the best of our knowledge, the AA centre is the only CLEAR centre that has 
explicitly identified how and why it will engage in different countries and the region. This is a 


88 Benin, Botswana, Burkina Faso, Burundi, Cameroon, DRC, Ethiopia, Ghana, Kenya, Lesotho, Malawi, 
Mozambique, Namibia, Nigeria, Rwanda, Zambia, Senegal, Sierra Leone, Swaziland, Tanzania, Uganda and 
Zimbabwe 
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positive and transparent process and realistic decision on its part that other CLEAR centres should 
consider. 


Efficiency 


Finding 30: The AA centre’s performance has been adversely affected by several 
inefficiencies. 


A review of the AA centre experiences over the period 2011-2014 indicates that several 
inefficiencies adversely affected its performance. These are discussed below. 


While the multi-donor trust fund established for CLEAR committed funding to the each of the 
CLEAR Regional Centres for a five-year period, the initial contract with the AA centre was for one 
year, and subsequent funding was linked to satisfactory performance.®? This approach was 
appropriate in view of standard trust fund procedures deriving from the need to manage risks on 
behalf of the donors. It was, however, not helpful in the context of CLEAR’s experimental nature 
which would take time to gestate. The initial short-term agreement contributed to a significant 
amount of repetitive administrative work (due to the need to prepare for and negotiate a new 
contract following the initial period) and contributed to employment insecurity for AA staff members. 


The establishment of the AA centre was adversely affected by delays from the Centre in submitting 
a project document with the appropriate structure and content to go through World Bank 
clearances; and, ii) delays from Wits in signing the grant, providing information on the authorized 
signatories, providing banking information and registering in the World Bank Systems to be able to 
access the funds. Furthermore, AA centre programme implementation has been negatively 
affected throughout the period: delays in the receipt of the CLEAR grant have created considerable 
cash flow issues for the centre and hampered its ability to implement activities and address staffing 
shortages.“The centre also experienced delays in identifying and appointing a Centre Director 
acceptable to the CLEAR Board. 


In addition there were often long delays (up to 5-6 months) between the conduct of an activity and 
reimbursement. This was reported to be particularly difficult for the cash-strapped affiliate centres 
(KSG and GIMPA), and acted as a disincentive for their involvement. While the World Bank made 
reimbursements within a week or two, Wits repeatedly took a long time to submit the required 
paperwork to be reimbursed. In Ghana, the Coordinator noted that this was contrary to earlier 
expectations of allocation of resources to implement work plan. In Kenya, KSG was not able to 
continue to pre-finance the AA centre activities for up to six months before being reimbursed and 
was thus not able to conduct activities. 


Interviewed AA centre staff reported that while the centre was quite successful in generating new 
sources of revenue, it lacked the ability’! to hire additional staff and thus effectively had twice the 
workload with the same staffing contingent). The centre decided to give priority to ‘fee for service’ 
activities over services provided with the subsidy grant. As a result, the AA centre did not carry out 





89 This arrangement was the same for all Centres. The multi donor trust fund established for CLEAR cannot 
make commitment of funds that it does not hold in cash. At the time the first grant was provided to AA, the 
TF did not hold sufficient funds to cover a longer period of time. For the second grant, the donors had 
already deposited additional funds into the TF account, so it was possible to make a 3 year grant. 


90 For example, at September 2014, the AA Centre reported that it is waiting to receive the second CLEAR 
grant. Wits has provided the Centre with an overdraft facility to accommodate its cash flow problems, without 
which the Centre reports it would not have been able to participate in the November 2013 Global Forum or 
the March 2014 AfREA conference among other activities. 


21 Due to various uncertainties including filling the Director's position and the uncertainties about the budget 
given the short-term nature of the initial CLEAR grant. 
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some planned activities or utilise the CLEAR grant as quickly as planned, nor did it realise some of 
its planned accomplishments due to its limited human resource capacities. 


Finally, since the person that filled the Centre Director position to February 2014 was not a Wits 
faculty member, he lacked the experience and contacts to navigate the university’s bureaucracy, 
and mitigate some of the delays normally associated with university actions and decision making. 


Together, these factors had (and continue to have) negative effects on the centre’s ability to carry 
out planned programming. 


Viability 


Finding 31: The AA centre has been effective in generating resources for M&E capacity 
building initiatives in Anglophone Africa. Nevertheless, the prospects for the 
centre’s viability are modest. 


To date, the AA centre has been quite successful in generating revenues from sources other than 
the World Bank. For example, in its first 1.5 years, it was able to generate USD 933,000 from other 
sources, which is roughly the same amount as the CLEAR grant. This is an impressive 
accomplishment, particularly given 

noted capacity and other challenges 


it faced during the period. In South Africa, DPME reimburses CLEAR for the costs of services 


Moreover, interview data suggest provided, and shares the costs of jointly developed new tools or 
that some potential AA national Services: 

government clients that are Other potential government partners in the region who were 
increasing their commitment to M&E interviewed (e.g., the Government of Zambia) indicated that they 
may have resources available would be willing and prepared to share costs of CLEAR services in 


(either from donors, or internal netune: 


resources) to offset the costs of 
M&E systems development. This will vary by country and by client within each country. 


While the AA centre has been quite successful in generating resources from various sources, the 
overall prospects for its viability appear modest, for several reasons. 


Integration in and support from host institution(s): The most significant reason relates to the 


relationship between the AA centre and Wits. On the one hand, the university reports that the post 
graduate diploma in M&E created as a result of the CLEAR project is fully institutionalized and self- 
sustained in Wits. Moreover, the university has played a critical financial role in helping the centre 
address its cash flow needs over the past 2.5 years, taking on financial risks in order to support the 
centre in getting established and implementing planned activities. On the other hand, the university 
has displayed limited ownership to date for the AA centre, which it regards as a temporary project. 
(Like many universities in South Africa and elsewhere, Wits has limited human and financial 
resources and many demands, so is obliged to manage resources strategically in keeping with its 
priorities. To receive greater attention and support from Wits, the AA centre would need to be able 
to generate visible benefits for the university in its core priority areas - enhancing its research and 
contributing to its student population. Those interviewed inside and outside the university associate 
the AA centre with an individual (the former Centre Director) rather than with Wits’ understanding 
of, commitment to, and ownership of the vision of CLEAR. 


Modest centre capacities: Centre staff turnover and capacity constraints have limited its ability to 
deliver services, meet the evolving needs of different clients, and play a more proactive role in the 
M&E community in AA. These factors further reduce the AA centre’s viability prospects. 


Planning for financial viability: The centre’s implicit business model has been to use third party 
subsidy (through the CLEAR grant) to carry out its activities, and complementing (and eventually 
replacing) these subsidies through revenues generated by centre activities (with considerable 
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success to date, as noted above). The AA centre has not yet developed an explicit sustainability or 
business plan and has not yet clarified where and how it intends to invest generated revenues. 


In conclusion, while the AA centre has done remarkably well in generating additional funding 
sources in its first few years, its current prospects for viability are constrained by a combination of 
weak internal capacities, and uncertainties about Wits support beyond its original commitment to 
2018. 


3.2.5 Conclusions and forward looking considerations 


The review of CLEAR AA’s performance to date identifies several very important contributions, 
most significantly to DPME, its important partner in the South African government, and to Wits 
University. The needs for, and relevance of the kinds of support it can provide is strong, and it has 
a positive reputation among its previous clients. Over the review period, the AA centre has met or 
exceeded most planned outputs and has been engaged in a variety of interesting relevant studies 
and initiatives in AA related to M&E. Finally, it has been successful in generating revenues from 
sources other than the World Bank. 


Nevertheless, among all Regional Centres, the AA centre has perhaps faced the greatest number 
and most significant operational challenges to date which have affected its performance. These 
include multiple inefficiencies related to finalising project documents, grant agreements and staffing 
the Centre Director position, considerable cash flow difficulties, issues with leadership of the centre 
and Wits, as well as staffing constraints — challenges which continue to seriously affect its 
performance today. Unless these challenges are addressed very soon by the CLEAR Board, World 
Bank and Wits University, it is not evident that the centre will be able to function in the future. As a 
consequence of its noted difficulties, it has had a modest presence and limited momentum with 
possible consequences for its credibility in the region. 


If the CLEAR Board, Wits University and the World Bank can resolve outstanding issues 
expeditiously, the CLEAR AA should consider the following suggestions: 


e Clarify and secure the institutional home for the centre within Wits University (the School of 
Governance or elsewhere) and identify ways and means to increase mutually beneficial 
collaboration for strategic and operational purposes. 


e Finalise RAC membership as required and set a time and agenda for the first RAC meeting 
as soon as possible. 


e With the input and support of the RAC and the Secretariat, review and revise the AA 
strategy and Theory of Change (to identify the learning questions that the centre will focus 
on) and the results framework (to clarify the measures of success that should be used to 
measure AA centre effectiveness and that results are realistic given the reduced calendar 
time remaining in the University’s agreement with the World Bank). 


e Contract or second the staff needed to align the centre’s managerial and professional 
capacities with its strategy. Ensure that the centre has ready access to staff/secondments 
who can navigate Wits University and World Bank requirements. 


e Complement centre staffing by nurturing working relationships with a broader pool of 
experienced partners and associates in the AA region to support strategy implementation, 
with particular attention to addressing needs in its four focus countries. 





e-Pact 49 


CLEAR Midterm Evaluation 





3.3 CLEAR Latin America (LA) Centre 
3.3.1 LA Centre Profile 


Selection and establishment 


The selection process for the (Spanish speaking) Latin America Centre (the LA centre) took eight 
months (April 2011-November 2011). The selected host institution was the Centro de Investigaciôn 
y Docencia Económicas (CIDE), Mexico, which was chosen out of 22 applicants from 7 countries. 
The LA centre became operational in February 2012 and was officially launched by Mexican 
President Felipe Calderon in June of the same year. 


Financial resources 


Total projected CLEAR funds for the LA Centre up to the end of FY 2018 (including grants, 
selection process, and Secretariat support) amount to USD 1,502,047.% The first CLEAR grant of 
USD 600,000 was awarded in May 2013, and received by the centre in May 2014. At the end of 
Q2, FY2014 the centre had not yet expensed any CLEAR funds but had operated on funding from 
other sources, in particular the Government of Mexico and CIDE. 


Staffing 


The LA centre staff has increased considerably since the centre became operational. Up to 2012, 
there were three full-time staff members (General Director, and Executive Director, and one project 
assistant).* At the time of data collection (February 2014), the LA centre had nine full-time staff 
positions: a Coordinator General, an Executive Coordinator, four project managers, an editor, a 
visiting professor, and a research assistant. 


Centre activities 


Since its establishment, the LA centre has Type of activity Feb. 2012 — 

developed a series of activities, services and Dec. 2013 

products as shown in the sidebar. In addition to Trairi m 

the noted events, the centre implemented Ee 

various internal capacity building activities for its Knowledge exchange 19 

staff. 

Centre clients have included representatives Technical assistance/advice 14 

from various national and sub-national , 

governments, non-government organisations, Collaboration witi MEE nelwarks 4 

academic and research organisations, Eee, easton 2 

development organisations, as well as M&E 

professional networks. Knowledge resource 6 
Grants, competitions 4 


92 Source: CLEAR Budget Update: Expenditures through FY2013 and Projections through FY2018. 
33 Source: CLEAR LA 2012 Annual Report. 
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3.3.2 LA Centre Context 


Finding 32: The LA centre was established in a regional context with considerable demand 
from national and sub-national governments for M&E and Performance 
Management services. The supply of M&E and PM expertise and related 
capacity building opportunities was inadequate, and M&E communities of 
practice lacked strength. This context has not changed significantly. 


The LA centre was created in a regional context in which many national governments had already 
incorporated M&E requirements in their legal frameworks and/or institutional practices, especially 
in relation to assessing social policy. However, the implementation and institutionalisation of 
managing for development results (MfDR) varied, with middle-income countries tending to be 
further advanced than lower-income countries.% 


The document review and interviewed respondents reported an increase in the demand for M&E 
since the 1990s which may be attributed to a number of reasons, including: 


e Overseas Development Assistance to Spanish speaking Latin America had significantly 
declined, making it more pressing to allocate resources to effective programmes only 


e Ongoing decentralisation processes in several Latin American countries (e.g. Argentina, 
Mexico, and Peru) had led to an increase in demand for monitoring and evaluation of 
public policies and social development programmes from sub-national entities. 


At the same time, the supply of M&E and RBM services in the region was inadequate in both 
quantitative and qualitative terms, and communities of practice lacked skills and coordination. 
Similarly, the number and quality of available M&E and PM capacity building opportunities were 
low, and there was a lack of context-specific learning and training materials in Spanish. 


This overall context has not changed significantly since the centre became operational, with the 
exception of changes in some (sub-)national authorities, e.g. in the state of Bacalar (Mexico) and in 
the Argentinean Cabinet which may — positively or negatively — influence the respective entity's 
interest in and demand for M&E and related capacity development. 


3.3.3 LA Centre Strategy and Capacity 
Strategy 


Finding 33: The LA centre adopted a phased strategy that increased its visibility in the 
region and made it an important provider of technical assistance and 
training/capacity building in M&E and PM. 


The LA centre’s strategy has evolved in a number of ways since its establishment. As emerged 
from both interviews with centre staff and the review of the original project document (2012), the 
initial objective of the centre was largely focused on positioning the LA centre as a regional M&E 
and PM knowledge hub. In order to do so, the centre adopted two main strategies: 


e The pursuit of partnerships or collaboration with multiple organisations in the 
region and in specific countries. At the regional level, these included the Latin American 
and the Caribbean Network of monitoring and evaluation and systematisation (RELAC), 
the Monitoring & Evaluation Network in Latin America & the Caribbean (REDLACME), 
the Latin America and Caribbean Community of Practice on Managing for Development 
Results (CoPLAC-GpRD), and the Inter-American Public Administration Education 
(INPAE). At the country level, the LA centre partners and collaborators included 


% Sources: IDB. Managing for Development Results. Progress and Challenges in Latin America and the 
Caribbean. the LA centre Project Document (2013), and Strategic Framework Planning Tool (2013). 
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universities such as the Universidad Catélica de Cérdoba (Argentina), Universidad 
Católica (Chile), and Universidad del Pacífico (Peru). 


e The sharing of experiences in institutionalising M&E and RBM practices in specific 
countries, such as Mexico or Chile, or at the sub-national level in various countries of 
the region. For instance, in 2012 the LA centre organised the International Workshop on 
Monitoring, Evaluation and Management for Results in Mexico City that included 
participants from sub-national governments from Argentina, Brazil, Chile, Costa Rica , 
El Salvador , Guatemala , Mexico , Paraguay , Peru, Dominican Republic and South 
Africa. The workshop was instrumental to gather their experiences in M&E and RBM 
that were disseminated through a CLEAR- CIDE publication. 


In a second phase, which started in mid-2013 and built on the benefits obtained from its increased 
visibility in the region, the LA centre's strategy has been geared towards responding to requests for 
technical assistance from national and sub-national governments and civil society organisations. 
This has created and strengthened the centre's relationships with public sector institutions in 
various countries. The centre responded to 14 such requests between July 2012 and December 
2013.% These requests were the result of the centre's previous efforts in facilitating the sharing of 
M&E and RBM practices, and included requests such as training and technical assistance on M&E 
and RBM provided to the Cabinet of Argentina following the international workshop in Mexico City. 
In addition, the LA centre has become an important actor for capacity building; 447 individuals from 
16 countries other than Mexico were trained by the LA centre as of February 2014. 


At the time of the field mission, the 
LA centre was in the process of 


further defining its niche by: LA Centre Outcomes as per Strategic Framework 2013 
i identifying specific sectors — Key government and civil society actors informed and effective 
requiring strengthening or raising users of results-oriented information 


awareness of M&E and RBM 


practices (such as public security 
and migration); and ii) engaging in Improved role of M&E and PM to inform decision making and public 
potentially innovative projects, such debate as an instrument to promote economic and social 


as a pilot project on social In 

technology and democratic 

governance with the Avina Foundation, so as to contribute to innovation in M&E. This does not 
mean that the centre stopped conducting activities to enhance its visibility in the region or 
responding to requests for technical assistance, but rather that the emphasis given to each 
strategy to achieve its stated outcomes (as identified in the CLEAR LA Strategic Framework, 
February 2013 (see sidebar) shifted over time. These outcomes have not yet been used to monitor 
or report on the LA centre achievements, given that the centre has been obliged to use the overall 
CLEAR results framework to do so. However, their main elements have been reflected in the type 
of activities that the centre has conducted, as well as in its choice of clients and partners. 


Improved legal and financial enabling environment for M&E and PM 


35 The LA centre does not record the number of requests received. 
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In alignment with its objective to influence national and sub-national M&E systems, the centre’s 
main client group has been national and sub-national public sector institutions in various countries 
(e.g. different ministries). However, because of differences in the country contexts, the LA centre 
has mostly provided technical assistance to middle-income countries (where M&E systems and 
expertise are more advanced), and 
training to low income countries (see ; TIF : : 
sidebar). In addition, the centre has Addressing demand for M&E services in low-income countries 
increasingly engaged with local and Because countries vary in their respective stage of development as 
regional networks, as well as with a regards M&E and PM, the centre’s strategy has been slightly 
number of universities in the region different in low-income countries such as Guatemala, El Salvador 
in order to strengthen the and Honduras. In these contexts, centre activities have focused on 
‘ ; i raising initial awareness among stakeholders (with focus on non- 
sustainable supply of high quality governmental organisations) by providing evidence of the 
M&E and RBM services. It has also importance of M&E and PM in the context of low-income countries. 
increased the number and diversity 


of topics that it addresses through its services. 
Human resources 


Finding 34: The LA centre has strong (internal and external) managerial and professional 
capacity. Nevertheless, it receives more requests for capacity building services 
than it can realistically satisfy given its existing human and financial resources. 


Managerial capacity: The LA centre has been able to build upon and benefit from CIDE’s 
managerial and administrative experience and systems. CIDE senior leadership are active 
contributors to the centre’s work, and provide strategic as well as managerial and professional 
advice. 


Professional capacity: The LA centre’s professional team has strong expertise in M&E and RBM in 
general and of the context for their application in the LA region. The vast majority of consulted LA 
centre clients and partners expressed appreciation of the team’s professional skills and described 
team members as dedicated. The centre also has access to additional capacity through CIDE staff 
and affiliates, and through partnerships the centre has established in Mexico and the region. 


Despite its considerable internal and external resources, according to consulted centre staff, the 
amount of requests for different types of capacity building services received from clients in the 
region far exceed the centre’s current capacity. This validates the centre’s strategy of seeking 
partnerships with other actors throughout the region (see below) and the development of a set of 
criteria to guide decision making on emerging opportunities in the future. 


96 Monitoring data up to Q2, FY2014 indicate that 38% of LA centre clients have been from government 
institutions, 10% from NGOs, 20% from academia. Another 32% were not specified. 
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Inter-institutional linkages 


Finding 35: The LA centre has established both formal and informal partnerships with a 
wide range of organisations in the region. This has strengthened its regional 
visibility, provided the centre with insights into existing needs of key actors, 
and allowed the centre to reach out to new actors. 


The LA centre has established formal and informal partnerships with a number of organisations in 
the region. The centre considers partnerships as strategic if they have the potential to enhance its 
regional networking and if the partner organisation has expertise in specific policy areas. 
Partnerships have been a key tool for the LA centre to enhance its visibility in the region and 
position itself at regional and country levels; gain insights into existing needs of key actors in other 
countries; and contribute to strengthening national capacity for M&E and/or RBM in these 
countries. Partnerships have also brought benefits to the LA centre in terms of generating 
additional opportunities for financial support/revenue generation, increasing the centre’s access to 
expertise available in other countries, and strengthening the centre’s credibility (due to its regional 
engagement) in the eyes of potential clients (especially national governments) and actual and 
potential donors. 


To date, the centre has established 


working relationships with other Praise for the LA centre cooperative attitude 

M&E capacity building providers, in During stakeholder consultations conducted for this evaluation, 
particular national and regional representatives from other organisations that provide M&E capacity 
networks of evaluators such as the building consistently acknowledged the LA centre for its willingness 
Latin American and the Caribbean and ability to collaborate with other service providers rather than 
Network of monitoring and creating competition. 


evaluation (REDLACME)) and 

individual RBM/M&E professionals. See sidebar. In addition it has collaborated with thematically 
focused organisations such as UN Women with whom it has explored the linkages of M&E, 
women’s empowerment and gender equality. The centre has formalised several partnerships 
through MOUs that identify the shared objective of the collaboration, as well as the types of 
activities that the partners will work on together. 


The LA centre’s partnerships have complemented its internal capacity by providing access to 
sector/theme-specific expertise and partners with knowledge of and connections in other 
geographic locations. 


Host institution arrangement and governance 


Finding 36: The host institution arrangement with CIDE has worked very well for the LA 
centre. 


Document review and consultations with CIDE and the LA centre staff provided evidence of 
significant benefits deriving from the centre’s host institution arrangement, and did not indicate any 
drawbacks or disadvantages related to the current set up. 


Since the LA centre was established, CIDE has provided resources to fund LA centre staff 
positions. CIDE faculty members are available to support centre activities, including research, 
training, and technical assistance. This has allowed the centre to respond more flexibly to requests 
for capacity building and/or technical assistance than it would have been able to with its internal 
staff only. Furthermore, the association with CIDE has positively influenced the reputation and 
credibility of the CLEAR centre in the eyes of potential clients, especially, but not limited to clients 
in the Mexican public sector. 


The centre is obliged to report annually to CIDE on its activities and must comply with CIDE rules 
and regulations on procurement and programming (e.g. as regards editorial/style standards for 
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publications, ethical and quality standards for the implementation of research projects, and fees for 
training activities). These regulations have, to our knowledge, not posed any conflicts in terms of 
compatibility with World Bank regulations, and consulted the LA centre staff agreed that the CIDE 
environment has been appropriate and uncomplicated in terms of its bureaucratic requirements. 


Finding 37: The LA centre has not yet established a Regional Advisory Committee. This 
has limited its ability to validate strategic decisions with relevant external 
actors. 


As noted in its project document (2012), the LA centre is committed to establishing a Regional 
Advisory Committee (RAC). While a functioning committee is not yet in place, two individuals have 
agreed to participate in the committee. Delays in the establishment of the RAC are attributable to 
challenges in finding experts that would not be in situations of conflict of interest. 


Consulted RA staff members expressed concerns over the fact that suggested RAC candidates 
have to be approved by the CLEAR board. In their view, this poses a risk of negatively affecting the 
centre’s ‘political capital’ in the region should the Board not approve one of the proposed RAC 
members. 


The absence of a RAC has limited the centre’s ability to discuss and thereby validate and 
legitimise its strategic plans and/or decisions with a broader set of relevant regional actors. 


3.3.4 LA Centre Performance 
Relevance 


Finding 38: While individual LA centre interventions have been relevant to the identified 
gaps in the region, it is not yet evident if or how these fit into longer-term 
strategies for the development of M&E systems at national, sub-national or 
sector levels. 


Document review and consultations with the LA centre staff, clients and partners indicate that, 
overall, the LA centre activities constitute a relevant and appropriate response to the existing 
capacity gaps and needs of the targeted Latin American countries and specific organisations. 
Specific strengths in this regard include the following. 


e Demand-driven and context appropriate: Centre staff has made efforts to tailor the 
content of capacity building interventions to the specific interests and needs of the 
respective target group. Client consultations indicated moderate room for improvement 
in this regard (e.g. several training participants expressed the desire for more case 
studies and examples of best practice related to their own context), but reflected an 
overall positive assessment of the centre’s efforts in this regard. The centre has put 
effort into the production, dissemination and use of context-specific knowledge and 
training tools in Spanish. 


e Differentiated: The centre has demonstrated its awareness of different needs, e.g. 
between middle-income and low-income countries in the region, with the latter requiring 
more basic M&E/RBM awareness-raising, knowledge and skills. 


While individual interventions have been relevant (in terms of their content, delivery mechanism, 
and client groups) to identified needs and gaps in the region, it is not yet consistently evident if or 
how these fit into broader and longer-term plans or strategies for strengthening M&E systems at 
national or sub-national levels. This may be due in part to the relatively short duration of centre 
operations, and more clarity may emerge over time, e.g. through longer-term collaboration with 
specific clients. However, the centre has not yet defined its own longer term vision of what 
‘success’ in different national or sub-national contexts is envisaged to look like, nor has it 
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developed a comprehensive Theory of Change to elaborate on how its work contributes to ongoing 
efforts to strengthen M&E systems at national, sub-national, or sector levels. 


Effectiveness 


Finding 39: At midterm, the LA centre has contributed to positive changes at the individual 


and organisational level in countries across the region. Some achievements 


have the potential to contribute to changes in national, sub-national, or sector- 


specific M&E systems. However the centre does not yet have systems in place 
to capture emerging longer-term results of its work. 


In approximately two and a half years the LA centre has conducted an impressive number of M&E 
capacity building activities that are in line with its evolving strategy and annual work programmes. 
As is to be expected at midterm, evidence of results deriving from these activities is largely located 
at the level of individuals and organisations.?* Nevertheless, some of the centre’s (planned or 
completed) activities and achievements do have visible potential to contribute to influencing 
national, sub-national, or sectoral M&E systems.® Selected examples are provided below.‘ 


National level: 


The LA centre conducted meta evaluations for various social programmes under the 
Mexican Social Development Ministry in Mexico. Evidence generated through these 
studies is expected to help inform the ministry’s future programmes. 


The government of Argentina requested technical assistance and training from the LA 
centre following an international seminar on M&E organised by the centre which had 
helped them identify various needs for strengthening systems and practices in their 
country. 


Sector level: 


The LA centre contributed to raising stakeholder awareness about the role of M&E in 
sectors other than social policies or programmes and created related networking 
opportunities. For example, a CIDE seminar on public safety, to which the LA centre 
contributed an M&E module, brought together government representatives, CSOs, 
experts in public safety, and evaluators. While it is too early to assess the specific 
results of this event, consulted participants noted that the seminar constituted an 
important milestone in engaging diverse actors in discussions. 


The centre supported the Peruvian ministry of women and vulnerable populations to 
create a monitoring index based on administrative and census information. The index 
will be used to assess how the ministry provides programmes and tracks results for 
vulnerable populations. 


Sub-national level: 


The centre entered into an agreement with the local government of Bacalar in Mexico to 


provide technical assistance for the conduct of a performance-based management 





9 Examples of such achievements in positively influencing awareness, knowledge and skills of individuals 
are provided in Annex O. 


98 Achievements that we consider to possess such “visible potential” are those conducted with individuals or 
organisations that are likely to influence others — be it due to their position and related power, or their status 
and reputation among peers. 


99 These also illustrate that centre activities have triggered additional demand both for M&E services and for 
related capacity building interventions. 
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diagnosis and compilation of a capacity development strategy. The diagnostic was 
completed in 2013. 


The LA centre assisted the state government of Jalisco in Mexico in its transition to a 
results-oriented M&E system, e.g. by developing indicators to measure progress of the 
state’s social programmes. 


The centre has made valuable contributions to strengthening the future supply of 
sustainable, high quality M&E services in the region. In particular, it developed modules 
for CIDE’s M&E postgraduate programmes (in public administration, migration, and 
public safety). 


Furthermore, in alignment with its envisaged strategy, the centre published a number of 
knowledge products, including a Performance-based Budgeting manual in Spanish, as 
well as the study “From Recommendations to Actions - Federal Programs Committed to 
the Evaluation Process (2011)” which explores the continued gaps in the actual use of 
evaluation findings for planning and decision-making in the Mexican government 
context. According to the LA centre’s data as of February 2014, the number of online 
consultations of these publications varied between 199 and 942 (depending on the 
publication). 


While LA centre achievements have significant potential to contribute to concrete (i.e. observable 
or measurable) changes in M&E systems, the LA centre (and CLEAR overall) has, until now, not 
had tools or mechanisms in place to track if and how progress towards such higher level results is 
being achieved. LA centre reports have tended to focus on completed activities. The centre has not 
conducted systematic follow-up activities with participants of capacity building events,*% nor has it 
systematically collected information on the (actual or potential) effects of interventions targeted at 
specific organisations (e.g. a particular ministry), or collected information on the frequency of use 
or quality of its knowledge products. In the absence of an explicit centre-specific Theory of 
Change, it is difficult for outsiders to position specific activities, partnerships, or achievements in 
the bigger picture of an envisaged pathway of change. 


Finding 40: The centre’s regional engagement has enhanced its visibility and credibility in 


the region and has created opportunities for increasing its financial autonomy. 
It is not clear if its regional engagement has contributed to more frequent and 
focused exchanges among policy makers and among M&E and RBM 
practitioners. 


Although the LA centre does not 


have an explicit regional strategy, it 
has consistently engaged with To date, the LA centre has organised at least 15 knowledge 


clients and partners from nine other 
countries in the region'™ primarily in 


exchange seminars, some of which have specifically focused on 
issues of regional interest. This included an international seminar on 
M & E country systems that was co-organised by the centre, the 


the form of technical assistance Argentine government" and the University of Buenos Aires. 
provided to national governments, 





100 A Tracer Study was conducted one year after the completion of the results-based budgeting workshop. In 
addition, the LA centre conducts participant evaluations following courses/training. The majority of 
participants indicated that there is a high probability that they would apply the knowledge/skills gained in their 
work. Some participants (though a minority) were either politicians, managers, or top leaders. 


101 Council of Ministries of the Presidency 


102 Argentina, Chile, Peru, Colombia, Brazil, Costa Rica, Panama, Uruguay, and El Salvador. In 2012, 66% 
of centre clients were from Latin American countries other than Mexico, and in 2013, 49%. According to 
centre reports, activities included participants from 14 countries in the region but these countries were not 


specified. 
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and in presentations and workshops it has given at regional conferences organised by 
organisations such as RELAC, the American Centre for Public Administration and Development 
(CLAD), and the Latin-American and the Caribbean Community of Practice on Managing for 
Development Results (CoPLAC-MfDR). 


Evidence derived from document review and stakeholder consultations indicates that the centre’s 
regional engagement has: 


e Enhanced the LA centre’s visibility in the region*% and helped the centre expand its 
insights into needs of key actors in other countries 


e Created additional opportunities for financial support/revenue generation, and increased 
the centre’s access to expertise available in other countries 


e Contributed to strengthening its credibility in the eyes of potential clients (especially 
national governments) as well as actual and potential donors. 


No evidence was found in the reviewed documents or through interviews of whether the centre’s 
regional engagement has contributed to more frequent and focused exchanges among policy 
makers or among M&E and RBM practitioners. 


The LA centre’s regional engagement has been facilitated by the fact that several countries have 
relatively advanced M&E systems, and are willing and able to share experiences related to the 
development and use of these systems. Having a common language has also been a positive 
factor. 


At the same time, working at the regional level has been challenging for the LA centre in terms of 
fully engaging with low-income countries such as Guatemala or Nicaragua where M&E needs are 
considerably different and more basic than in countries such as Mexico or Chile. At a practical 
level, the ability of the LA centre to reach out to the whole region (beyond involving individuals from 
other countries in training programmes or other events) is limited, given the size of its team, and 
the availability of potentially strategic partners in some (especially less advanced) countries. 


Efficiency 


Finding 41: The CLEAR LA centre has been successful in leveraging funding from other 
sources than CLEAR. 


Experience to date reflects the LA centre’s considerable ability to attract funding from a variety of 
sources, as well as the conducive national and organizational environments in which it is operating. 
By the end of 2013 the LA centre had implemented the highest number of capacity building 
activities among the five existing CLEAR centres. All of its activities up to that point had been 
funded by sources other than CLEAR, or through self-generated income from paid services.' The 
main donors that the centre was able to leverage were the Mexican Government! (USD 500,000) 
and the Bill and Melinda Gates Foundation grant (USD 186,395). In addition, CIDE consistently 
supported the centre by funding staff positions. 





103 Knowledge/visibility of the LA centre is still concentrated in Mexico, as recognised by the LA centre staff 
and as emerging from data from Google Analytics (for the period 1/4/2012 to 6/2/2014) showing that over 
70% of individuals who had consulted the LA centre’s website were from Mexico. Nonetheless, available 
data from Google Analytics, Twitter, and Facebook suggest that there has been an increase, from 2012 to 
2013, in the LA centre’s visibility outside Mexico. In addition, the centre has received an increased number of 
requests for technical assistance from countries other than Mexico. 


104 According to the CLEAR LA budget, the first activities to be funded through the CLEAR grant will relate to 
the establishment of its Regional Advisory Committee. At the time of writing these have not been completed. 


105 In particular the Ministry of Finance, and the Ministry of Governance, with whom the centre has a 6-year 
collaboration agreement. 
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The first CLEAR grant of USD 595,000*% awarded to the LA centre in May 2013 but was only 
received in May 2014. This was due to the time needed to ensure that policies and procedures of 
CIDE complied with World Bank procurement and financial management requirements, as well as 
to delays in grant signature by the Government of Mexico.*% The CLEAR grant was ready to be 
disbursed as of May 2013. However, at that time the LA centre requested that the Secretariat 
postpone disbursement until early 2014. This was based on the fact that receiving the grant late in 
the year would have implied considerable administrative challenges, e.g. related to reporting and 
audit requirements. Given the substantial resources that the centre had been able to leverage from 
other sources delaying accessing the CLEAR grant did not negatively affect its activities. 


Viability 


Finding 42: There is a strong likelihood that the LA centre and its work will be sustained 
after CLEAR funding ends. 


This assessment is based on the following factors: 


Planning for financial viability: The LA centre team is dedicated to the purpose and work of the 
centre and has developed a plan for financial sustainability. In this document the centre commits to 
ensuring programmatic impact without sacrificing financial viability by applying a strategy of cross 
subsidies between service lines as well as among customers with different financial capabilities. 
The centre has also established an endowment fund within a CIDE trust to administer revenues 
generated through the centre's activities. LA centre staff report that the centre has already 
generated approximately 

USD 300,000 for the endowment. 


Integration in and support from host To date, most of the LA centre's activities have been financed 

institution(s): CLEAR's work is well- through sources other than the CLEAR grant. Based on this, the end 

aligned with CIDE's mandate to of CLEAR funding is likely to constitute a less notable change than it 
may for other CLEAR centres that have relied more strongly on 


advance scientific knowledge and CLEAR funding. 
improve decision making. In | | 
addition, CLEAR is seen as a However, some interviewed stakeholders see the LA centre’s 


capacity to sustain itself in the long term as being closely linked to 


valuable addition to the work of the existence of the global CLEAR initiative. In their view, one 


CIDE in that it provides an important characteristic of the LA centre that sets it apart from other 
additional regional perspective and M&E capacity building providers in the region is the fact that it is 
related networks. CIDE has seen to be linked to a broader, internationally supported initiative. 


included CLEAR as a strategic 

project in its Medium Term Strategy.*% There is strong collaboration among CLEAR and CIDE 
staff, and centre activities are supported by CIDE administrative and professional staff, as well as 
through the network of CIDE associates. In addition, CIDE has consistently committed resources to 
fund the LA centre staff, and is committed to continue doing so. See also sidebar. 


Context: Demand for M&E services and related capacity development and technical assistance 
remain high among national and sub-national governments in the region. In addition, the 
experience of the LA centre to date has shown that a considerable number of stakeholders are 
both willing and able to allocate resources for such services. For example, the government of 
Mexico has already demonstrated its interest in and commitment to providing financial resources 


106 Overall CLEAR resources expensed or committed to the LA centre through FY2013 are the second 
lowest among the existing CLEAR centres. Only the East Asia centre has received less. 


107 The administration had just changed at the time, and the head of office in charge of the signatures has 
not yet been appointed. 


108 Source: CLEAR LAC Monitoring Tool. 2012-2013. 
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for M&E/RBM capacity building. It is likely that it will continue to do so, either through the CLEAR 
centre as a separate entity, or by supporting a continuation of parts of CLEAR programming 
through CIDE. 


3.3.5 Forward looking considerations 


The following suggestions for consideration by the LA centre derive from the findings outlined 


above. 
1. 


Develop a theory of change (ToC) for the centre (or country-specific theories) to help 
external stakeholders understand how individual centre activities are envisaged to 
contribute to system level changes. 


Develop (or strengthen) monitoring tools and processes to capture emerging, longer term 
results at individual, organisational, sector, and system levels. These should be closely 
linked to the ToC in terms of defining specific progress markers: What will ‘success’ in each 
country/in the region look like? 


Review and decide on future approach to regional engagement. For example, does the LA 
centre want to reach out more systematically to low-income countries, or does it want to 
focus on the relatively easier targets that it has worked with to date? Either way is fine, e.g. 
latter option might allow the centre to focus more strongly on research/learning, which 
might benefit less developed countries in the longer term. 


Find additional mechanisms that go beyond participation in regional events that can further 
strengthen the centre's regional perspective and knowledge of emerging opportunities, 
actors, and needs in the region. 


Continue to push ahead with the establishment of a functional Regional Advisory 
Committee that can provide informed, yet slightly distanced insights and suggestions to the 
centre as regards its priorities and approaches. 


Further explore and capture emerging lessons on what it takes to recognise, influence, and 
use momentum for engaging with various types of public sector institutions in different 
countries. 
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3.4 CLEAR South Asia (SA) Centre 
3.4.1 SA Centre Profile 


Selection and establishment 


The Abdul Latif Jameel Poverty 

Action Lab (J-PAL) South Asia, Both J-PAL and CLEAR are centres within IFMR LEAD which 
located within the Institute for comprises five independent research centres, each of which has its 
Financial Management and own mission but is expected to contribute to a common vision.'°° 


Research (IFMR) in Chennai, Tamil J-PAL SA is fiscally and otherwise responsible for the CLEAR centre 
Nadu, India, was selected from 24 in South Asia and uses IFMR rules and procedures for procurement 
applicants from six countries in and human resources. 

South Asia to serve as the CLEAR 

regional centre in December 2010. See sidebar. The centre became operational in September 
2011. 


J-PAL is partnering with the Center for Economic Research, Pakistan (CERP) in Pakistan to serve 
as a CLEAR affiliate centre and support the aim to work across national borders in the region. A 
Regional Advisory Committee (RAC) has been established to provide guidance to the centre. 


Financial resources 


The first grant agreement between the World Bank and IFMR (as the host of J-PAL SA) was 
signed in August 2011 and the grant of USD 640,000 was received by the centre in September 
2011. A second grant for USD 1,300,000 was committed by the World Bank in October 2012 and 
signed in July 2013.''° According to its annual reports, the centre expended USD 504,178 during 
the period 2011 to July 2013, and the total budget for completed activities during Q1 and 2, 2014, 
was USD 115,356. Total projected funds for the South Asia centre up to the end of FY 2018 
(including grants, selection process, and Secretariat support) amount to USD 3,202,364. 


Staffing 


J-PAL: The size and composition of the CLEAR team at J-PAL has not changed significantly over 
time. When the centre was established, it had six full-time and four part-time staff members. At the 
time of writing (July 2014), it had five full-time staff members (Programme Director, Capacity 
Building Manager, Events and Communications Manager, Capacity Building Associate, Events and 
Communications Assistant) and four part-time staff (Capacity Building Manager, Deputy Director- 
JPSA, Executive Director JPSA, and an Admin & Operations Manager). 


CERP: The CLEAR team at CERP? has one full-time person (Policy and Training Associate) and 
three part-time staff (Associate Director; Admin Manager, and Communications Assistant. During 
the early stages of centre establishment all of these positions were full-time to support the setting 
up of systems and processes. 





109 IFMR Lead is “dedicated to leveraging evidence-based research to further economic and financial 
development of poor people living in India and other low-to-middle income countries.”.” Source: 
http://ifmrlead.org/ 


110 Sources: CLEAR South Asia Demand Assessment; CLEAR (global) Annual Report July 2012-2013, 
CLEAR South Asia Annual Report 2011-2012. 
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Centre activities 


Since mid-2011, CLEAR South Asia has Type of activity June 2011 to 
implemented a considerable number of December 2013 


capacity building interventions, as is illustrated 


in the sidebar. In addition, centre staff provided  l'aining 28 
more than 100 days of advisory service to Pe an een E 
various government and non-government £ £ 

clients and implemented approximately 30 ENEE 7 


internal capacity building activities for its own 
staff. Collaboration with M&E networks 9 


From the onset, centre clients included 
representatives from government (national and 
state level), national and international non- 
government organisations, donor agencies, 
academic and research organisations, as well 
as from M&E professional networks. 


3.4.2 SA Centre Context 


Diagnostic 1 


Knowledge resource 4 


Finding 43: When the SA centre was established in 2011, regional stakeholders had shown 
an interest in and commitment to RBM and M&E but there was very limited 
supply of high quality M&E services and related capacity building opportunities 
and demand for high quality evaluations was not yet broad-based. This context 
has not significantly changed. 


Key characteristics of the regional context in South Asia at the time the CLEAR centre was 
established!" include the following: 


e Across countries there were individual champions of reform with an interest in 
institutionalising M&E, but this interest was not yet broad-based, nor was it supported 
through strong M&E expertise and capacities on the supply side. Overall, the culture of 
evaluation and evidence-based decision making in the region was weak. 


e M&E capacity building services in the region were scarce and concentrated in a few, 
often pre-service programmes. Available in-service training in M&E for practitioners 
(including government officials) tended to be short one-off sessions that were not widely 
applicable. As a result, the quality of analysis and evaluation generally remained poor 
and focused on expenditure tracking rather than on insightful evaluations. 


e In some countries (e.g. India and Bangladesh) civil society organisations showed an 
increasing interest in monitoring the performance of government actors and holding 
them accountable. However, NGOs and other organisations often lacked the necessary 
M&E capacity to fulfil evidence-based advocacy or ‘watchdog’ functions.''* There was 
active demand from NGOs for capacity building and technical assistance in this regard. 





111 As described in CLEAR SA proposal (including original demand assessment), project documents and 
annual reports, and confirmed by literature review and stakeholder consultations. 


112 One consulted national partner of the CLEAR SA centre noted that “Here, when civil society or media 
demand accountability from the government, what they usually mean is ‘prove that you didn’t steal from us’ 
as opposed to ‘show us results’. But it’s a process, and there is definitely movement in people being 
interested in what the government does and achieves with the available resources.” 
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e The M&E community in South Asia was not yet well organised, with only emerging 
communities of practice at national and regional levels. Knowledge sharing, advocacy 
and interaction among these actors were in early stages. 


While the types of existing opportunities, needs and challenges for M&E were similar across the 
region, they manifested differently in each country.''? Overall, implementation risks in the region 
were low to medium, with the most significant risks being posed by political volatility (in Pakistan 
and, to lesser extent, in Bangladesh), upcoming elections in India, and frequent government 
turnover in various countries. With the exception of the (completed) Indian elections, this risk 
assessment remains valid. 


3.4.3 SA Centre Strategy and Capacity 
Strategy 


Finding 44: The SA centre has used a dual approach, combining M&E capacity building 
with policy outreach in order to strengthen both the supply of and demand for 
well-implemented monitoring and evaluation services. Some elements of its 
strategy have evolved based on implementation experience. 


From the beginning, and as reflected in the types of activities it has carried out, the SA centre has 
combined efforts to strengthen M&E-related knowledge and skills with initiatives to raise interest 
and awareness of the potential benefits of high quality M&E among various stakeholders. The 
centre’s overarching goal has consistently been to strengthen national capacity required for 
implementing evidence-based policies. 


While these key elements have remained the same, other aspects of the centre’s strategy have 
evolved based on experience gained from implementation. Some of these are outlined below. 


e During its first year, the centre’s programming was primarily ad hoc and opportunistic, 
e.g. by offering a variety of one-off open enrolment courses to a variety of government 
and non-government stakeholders. This approach was appropriate for the initial stages 
as it helped the centre “put itself on the map”. Since 2012, the SA centre has 
increasingly provided custom-made capacity building services for individual 
organisations in order to meet the specific needs of their staff and managers, and 
engage in longer term relationships with clients. 


e One idea introduced in the (Draft) Project Document (October 2012-July 2015) is the 
intent to develop demonstration/pilot programmes and capacity building activities with 
the ultimate intent to influence state or national level policy. The implicit underlying 
assumption is that creating pockets of good practice will contribute to influencing other 
actors in the longer term. 


e The 2012-2015 Project Document also put stronger emphasis on the centre’s intent to 
engage not only with actors at that national level, but also work to strengthen capacity 
at sub-national and local levels to support bottom-up processes for monitoring and 
accountability. 


To bridge gaps between concrete activities and the high level outcomes outlined in the overall 
CLEAR Theory of Change, the SA centre is in the process of developing an internal results 
framework to define region-specific intermediate outcomes. 


113 Please see Annex N for further details on the national contexts of India and Pakistan. 
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Human resources 


Finding 45: Although the SA centre has strong internal and external managerial and 
professional capacity, it receives more requests for capacity building services 
than it can realistically satisfy given its human and financial resources. 


Managerial capacity: The centre has had strong managerial capacity, due in part to J-PAL’s 
managerial and administrative experience and systems. The J-PAL South Asia Executive Director 
and Deputy Director are active contributors to its work, and provide strategic as well as managerial 
and professional advice. The current team CLEAR staff includes full-time positions for an Events 
and Communications Manager and Assistant, as well as for a part-time Admin and Operations 
Manager. This has freed other staff to focus more on programming. 


Professional capacity: The professional centre team has strong expertise in impact evaluation and 
quantitative approaches, as well as, more generally, in the theory and practice of ensuring rigour in 
data collection and analysis. Individual team members also have experience with more qualitative 
approaches. Consulted CLEAR SA clients and partners unanimously described the centre team as 
skilled, energetic, dynamic, passionate, dedicated, hard-working, and pleasant to interact with. In 
addition, the centre has access to a rich pool of external capacity that it can draw upon for advice 
and hands-on assistance in the planning or implementation of capacity development interventions. 
It has been able to tap into existing J-PAL, IFMR, and CERP staff and associates in the region, as 
well as globally. The SA centre has established agreements with other organisations such as 
Innovations for Poverty Action (IPA) in Bangladesh with whom it collaborates to plan and deliver 
services. 


Despite these considerable internal and external resources, consulted centre staff noted that the 
number of requests for its services from clients in India and other countries in the region far 
exceeds the centre’s capacity. This contributes to the centre’s intent to focus its work more 
narrowly in the future — as discussed below. 


Inter-institutional linkages 


Finding 46: The SA centre has established both formal and informal partnerships with a 
wide range of organisations in India and the region. To date, many of these 
partnerships have tended to be activity focused rather than strategic and long- 
term. 


The CLEAR SA centre has established a range of both formal and informal partnerships with a 
number of organisations in India, and also in other countries in the region. Beyond those 
mentioned in the previous finding, these include partnerships with sectorally or otherwise 
specialised actors with whom the centre collaborates on specific activities, e.g. with UN Women on 
a series of roundtable discussions on the linkages of gender equality and evaluation. Other 
partners have included NGO actors such as the influential Indian NGO Pratham, as well as the 
World Bank office in India. The latter in particular has been valuable in identifying and making 
available resources for promising opportunities such as upcoming efforts to support several state 
level governments in India in strengthening their M&E systems. 


To date, partnerships that CLEAR SA has engaged in (especially those outside of India) have not 
yet established a common vision of the ‘bigger picture’ changes to which the organisations can 
jointly contribute. In our experience, this is common during early stages of partnerships. While 
consulted CLEAR partner organisations in India, Sri Lanka and Bangladesh expressed their 
appreciation of the collaboration with CLEAR, almost all of them also noted an interest in taking the 
partnership to a higher level by engaging with CLEAR in longer term and strategic thinking that 
would go beyond planning specific, short term activities within their own organisation. This interest 
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may provide valuable opportunities for future engagement, especially since the SA centre is not in 
a position (given its resources) to maintain a permanent presence outside of India.*** 


Host institution arrangement and governance 


Finding 47: The host institution arrangement has posed some modest challenges, but has 
worked very well overall for the CLEAR SA centre. 


Among the existing CLEAR centres, the South Asia centre is unique in that it is hosted by an 
organisation (J-PAL) that is itself hosted by another institution (IFMR). On the positive side, this 
arrangement has given the CLEAR SA centre access to a wide pool of associates and contacts 
provided through the two host institutions. It has also meant, however, that the CLEAR SA centre 
must report to two host institutions in addition to CLEAR donors. However, evidence derived from 
document review and stakeholder consultations indicate that this has not negatively affected the 
centre's performance. 


The CLEAR SA centre director has the status of a programme director in J-PAL, and CLEAR staff 
is part of the wider J-PAL SA Policy team. The director of IFMR chairs CLEAR SA centre's 
Regional Advisory Committee (RAC). Similarly, the (part time) Associate Director in Pakistan is a 
staff member of CERP, which contributes to the integration of the centre within its host institution. 
While the CLEAR team is considered autonomous as regards strategy development and decision 
making, it has to take into account strategic advice received from J-PAL, IFMR, CERP, its RAC, 
and the CLEAR Secretariat. Consulted SA centre staff noted that in their view this advice was 
sometimes contradictory. Under the current grant arrangement, the Secretariat (and not J-PAL or 
the SA centre) has approved new programming endeavours of considerable size.''® 


Finding 48: The SA centre was the first and, until now, the only CLEAR centre that has 
established a functioning Regional Advisory Committee (RAC). The RAC has 
provided the centre with informed and appreciated feedback and advice. 


The RAC was established in November 2012 and to date has met twice (February 2013 and 
January 2014). Four of the five current RAC members are based in India and the centre is 
exploring how to broaden RAC membership to ensure better regional representation. The sensitive 
political relationship between Pakistan and India, and related travel restrictions, has posed some 
challenges. The RAC composition is expected to be reviewed in line with the evolving future role 
that the centre intends to play outside of India. 


Consulted SA centre staff and RAC members have been highly satisfied with the role and 
functioning of the RAC to date. The group has provided constructive criticism and practical 
suggestions on how the centre can further strengthen its approach to make the best use of its 
available resources. RAC members have been able to draw upon their in-depth knowledge of the 
regional and national (especially Indian) contexts to provide insights on existing needs and gaps 
and likely pathways to change. Some RAC members have offered to be available to centre staff on 
an informal and ad hoc basis to provide advice on specific programming decisions. 





114 With the exception of Pakistan, although CLEAR presence in that country is also limited — It should also 
be noted that CLEAR's current partner organisations in other countries tend to have limited human and 
financial resources to deliver M&E capacity development activities or to be pro-active in their relationship 
with the CLEAR SA centre. 


115 While dollar thresholds of projects requiring clearance are provided in grant agreements, this has been a 
topic of ongoing discussions between the Secretariat and the CLEAR SA centre team. At the time of writing, 
two projects that are awaiting Secretariat approval have budgets of USD 30,000 (Technology-Enabled M&E) 
and USD 80,000 (collaboration with ASER in grassroots level M&E capacity building) respectively. 
Secretariat approvals relate to the nature of contracting appropriate for the respective contract, i.e. sole- 
source or a competitive bid, in order to follow procurement rules. 
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3.4.4 SA Centre Performance 
Relevance 


Finding 49: SA centre activities have been relevant in view of the identified gaps in the 
supply of and demand for high quality M&E services in the region. 


Overall, the types and combinations of interventions chosen, the types of actors targeted, and the 
topics addressed by SA centre made sense in view of the noted gaps in awareness, knowledge, 
skills as well as systems and broader evaluative culture (see context section) that the centre was 
trying to address.''® Key strengths in this regard include the following: 


e The SA centre deliberately engaged with a broad variety of stakeholders (government, 
non-government, academia, donors) at both national and sub-national levels. This was 
appropriate in view of the centre’s underlying assumption that sustainable change in 
M&E systems would require both top-down and bottom-up improvements. 


e Supply side: Centre activities included both broad based and targeted capacity building 
for individuals and units likely to conduct or manage evaluations, thereby aiming to 
enhance their awareness, knowledge and skills necessary for engaging in high quality 
M&E services. The centre has also made efforts to support networking and exchange 
among evaluation 
professionals at national 


i 117 
and regional levels. Efforts focused on influencing demand for M&E services 


e Demand side: Supply- In India, the CLEAR SA centre, in collaboration with other 
focused efforts were organisations, organised a series of 10 roundtable discussions on 
complemented by various M&E related topics. The events were attended by 
interventions aiming to approximately 200 people and by 900 viewers online. Consulted 


stakeholders in India expressed their appreciation for the events, 
which they saw as valuable and innovative opportunities for 
knowledge sharing, awareness building and networking among 


influence the awareness, 
willingness and interest 


of various kinds of diverse actors involved in evaluation. Centre staff has also 
decision makers (see conducted various one-on-one meetings with high level decision 
examples in sidebar). makers in India to raise their interest in and awareness of the 
Awareness-raising was benefits of strong M&E. 

particularly relevant in In Pakistan, the CLEAR SA centre facilitated a Policy Dialogue on 
organisational or Education Support Services, putting special emphasis on backing 
national contexts where interventions by data-driven evidence and building effective M&E 


there was limited interest | systems. 
in and awareness of the 
benefits of M&E. 


Many of the SA centre activities are not strictly focused on either supply or demand, but have the 
potential to influence both dimensions. For example, CLEAR SA centre staff reported that training 
activities have led client organisations to later approach the centre for more targeted support, 
thereby demonstrating changes in the internal demand for higher quality M&E and related capacity 
building. 





116 Several stakeholders in Pakistan expressed some concern that the content of some capacity building 
offers has been slightly above the existing capacity of national stakeholders. For example, they questioned 
whether a demanding topic such as Impact Evaluation was the most pressing issue to be addressed in a 
context such as Pakistan. However, consulted training participants noted that the topic had been relevant 
and useful to them. 


117 For example, with organisations such as the Sri Lanka Evaluation Association (SLEVA), the Community of 
Evaluators (CoE) South Asia, and the CoE Pakistan. 
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Effectiveness 


Finding 50: At midterm, the CLEAR SA centre has engaged with an impressive list of 
diverse clients and has contributed to positive changes at the level of 
individuals and some organisations. While some of these changes have 
potential to positively influence M&E systems at national, sub-national, or 
sectoral levels, it is not yet consistently possible to measure that and how the 
sum of individual achievements are likely to add up to broader or higher level 
results. 


After slightly less than three years A 
of operations, the CLEAR SA centre | Measuring Results 


has implemented an impressive To date, all information on results is based on self-reporting from 
number of M&E capacity building targeted beneficiaries. For example, the SA centre has solicited 
activities in line with its evolving client feedback at the end of training events and has received 


informal feedback from individual clients on other capacity building 
modalities. Other information on results achieved to date derives 
from consultations with SA centre clients during this evaluation. 


strategy and related work 
programmes. To date, there is 
some evidence of results, (i.e. of 
actual influence on knowledge, 
skills, attitudes, practices, or 


Capturing the effects of advocacy efforts is difficult under any 
circumstance. Nevertheless, while SA centre policy dialogue events 
have not yet led to concrete results such as actionable plans, 


culture) at the level of individuals consulted participants noted that these events had provided 

and some organisations.''® While opportunities for discussions among policy makers, advocates, and 
this is to be expected at midterm, practitioners. 

the SA centre (and CLEAR overall) To strengthen its ability to capture longer term effects of its work, the 
has not had tools or processes in CLEAR SA centre is in the process of developing an internal 

place to monitor or measure its monitoring plan to accompany its internal results framework. 


contributions to the longer term 
and/or higher level results of its work. See sidebar. 


Nevertheless, a number of achievements have the visible potential to contribute to influencing M&E 
systems (in terms of influencing M&E standards, tools, and practices) beyond the client 
organisation or agency.''? For example: 


e National M&E Office: The CLEAR SA centre established a partnership with the India 
Independent Evaluation Office (IEO) soon after its creation in 2013. The CLEAR SA 
centre, in collaboration with the LA centre, organised an exchange between the IEO 
and CONEVAL from Mexico. Insights from this exchange led the IEO director to 
suggest changes to the structure of the IEO in a note to cabinet. Supporting the IEO is 
the most visibly strategic partnership that the SA centre has engaged in to date in view 
of its potential to eventually effect system-wide changes at the national level. IEO was 
disbanded September 2014 


e Federal Government: In 2013, the SA centre organised training workshops on impact 
evaluation for Indian Economic Services (IES) Probation Officers and mid-level officers. 
The workshops focused on the use of impact evaluation to inform policymaking and 
programme design, and addressed survey design and data collection. The SA centre 
hopes that by addressing probation officers who are at the beginning of their career, the 
training will contribute to increased M&E awareness and basic skills across the IES 


118 Examples of such achievements are included in Annex O. 


119 Achievements that we consider to possess “visible potential” are those conducted with individuals or 
organisations that are likely to influence others, due to their position and power or their reputation among 
peers. Influence may be exercised by demonstrating good practice or by having authority to define and 
implement standards, guidelines, and other frameworks that guide the M&E work of others. 
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staff, and in turn contribute to demand for more and higher quality M&E. There is no 
information available yet on whether or how participants have used the knowledge and 
skills acquired in the training, and it is too early to expect changes in the broader 
organisational culture and related practices. 


e Sub-national level governments. Through training and advisory services, the SA 
centre has engaged with a variety of state level governments in India, including the 
governments of Tamil Nadu and Haryana. For example, the centre provided technical 
assistance to the Centre for Research and Experiments for Action and Policy (REAP) 
within the Government of Haryana’s Education Department to oversee the M&E of on- 
going projects!?” and eventually to inform education policies. Consulted REAP staff 
appreciated the support from the SA centre, but noted that it was too early to assess 
what effects this assistance will have on the quality of REAP’s M&E work. '?' 


e Donor agency: The CLEAR SA centre won a competitive bid issued by the USAID 
office in India to help the agency integrate impact evaluation for learning and 
accountability. The centre developed an impact evaluation toolkit and worked with 
technical teams within the agency to strengthen their understanding and skills related to 
planning for and managing impact evaluations of USAID-supported initiatives. It is too 
early to assess whether and how USAID will reflect this capacity building work in its 
future practice of commissioning and managing evaluations, and, moreover, if and how 
the agency will share related experiences with others, and with what effects. 


e NGO: The SA centre worked with Breakthrough, an international human rights 
organisation, to institutionalise and improve its M&E processes, in particular its ability to 
plan for and manage an upcoming impact evaluation of one of its projects using a 
Randomised Control Trial (RCT). The NGO intends to share related experiences with 
other, especially non-government actors. 


One core idea of the current CLEAR SA strategy is to create examples of good practice at various 
levels that it hopes will have a snowball effect. This is reasonable in light of the magnitude of 
existing gaps and needs and the limited influence that any modestly resourced centre can have. 
However, the CLEAR SA centre has not yet made explicit how very individual examples of good 
practice will come together and influence each other, or whether each of these examples is 
regarded as a more or less stand-alone pilot that could be replicated with similar actors in different 
settings but that are not expected to influence each other. This uncertainty contributes to the 
overall impression of individually relevant and promising results that are fragmented and do not yet 
clearly add up to a consistent whole. 


This observation was shared by most consulted CLEAR SA partners and has been noted by its 
RAC. The SA centre has already expressed the intent to sharpen the focus of its future work, but 
has not yet defined criteria that would help decide whether a specific partner, type of intervention, 
or role(s) played by the centre is strategic and why. The SA centre’s most recent draft strategy 
(2013-2015) and draft internal results framework still outline a wide range of services, clients, and 
thematic areas; it is not yet evident how the different components are seen to complement each 
other, and to what end. 


Finding 51: The SA centre has made valued, but fragmented efforts to work with other 
countries in the region. This reflects that CLEAR overall has not adequately 


120 Such as a Midday Meal Scheme, teacher training, and Meena Manch (Adolescent Girl Clubs) 


121 The centre is also planning to work with several other state level governments in India to assess and help 
improve their monitoring systems. It is hoped this will provide positive examples that can be shared with 
other state level governments interested in implementing similar changes. 
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defined its expectations of how Regional Centres should engage in a regional 


approach. 


While most CLEAR SA centre activities have focused on India and, to a lesser extent, Pakistan, 
the centre has continuously engaged with individuals and organisations from other countries in the 


region (Bangladesh, Sri Lanka, 
Nepal) to help strengthen their 
individual or collective capacities. 


The SA centre’s regional approach 
has evolved over time, as is shown 
in the sidebar. To date, however, its 
regional engagement has been 
fragmented, and the evaluation did 
not find the same kind of 
strategically promising interventions 
or partnerships as were noted in 
India. 


In our view, this does not point to a 
shortcoming of the centre’s work, 
but rather to a conceptual 
uncertainty within the overall 
CLEAR initiative about the purpose 
of regional approaches and realistic 
timeframes for working at the 
regional level. This includes a lack 
of shared understanding within 
CLEAR overall of what the notion of 
‘region’ refers to.'22 


The expectation that all CLEAR 


Evolution of CLEAR SA Centre’s Regional Approach 


The SA centre’s initial idea was to identify partner organisations in 
other countries that would take the lead in conducting capacity 
development activities either on behalf of or in collaboration with 
CLEAR. However, with the exception of Innovations for Poverty 
Action (IPA) in Bangladesh, the capacities of potential partner 
organisations were not strong enough to meet the centre’s quality 
expectations. 


Recent efforts have instead focused on: 


a) Identifying actors in other countries who are interested in 
strengthening their own capacities and who may serve as role 
model for other actors (e.g. BRAC) 


b) Engaging with existing/evolving communities of practice, e.g. the 
Sri Lanka Evaluation Association (SLEVA), and supporting their 
capacity building events 


c) Collaborating with regional actors that aim to influence the supply 
side of M&E, e.g. the Teaching Evaluation in South Asia (TESA) 
initiative. 


In future, following advice from it RAC, the SA centre intends to 
focus its work on India, Pakistan, Bangladesh, and Sri Lanka to 
make the best use of its limited resources. It is currently developing 
a Regional Strategy to make this explicit. 


centres were to engage with actors outside of their home country has put additional strain on the 
limited CLEAR SA staff and financial resources without being guided by a shared understanding of 
the intended benefits and purposes of this regional engagement. To date, implementation 
experience has demonstrated the SA centre’s willingness to comply with related expectations, but 
it has not yet generated strong evidence supporting the underlying CLEAR design assumption that 
working at the regional level would be beneficial in facilitating change in each country or across 


countries. 
Efficiency 


Finding 52: With the exception of some delays in the transfer of funds in 2013, no concerns 
were raised as regards the efficient use of CLEAR resources funds in and 
through the South Asia centre. 


Timely transfer of funds 


In 2013 the SA centre experienced a considerable delay of about nine months in the release of 
CLEAR funds. While J-PAL was able to provide bridge funding which allowed activities in India to 


122 Some consulted CLEAR centre staff members (not limited to South Asia) assumed that the expectation 
was for each centre to work in, or try to ‘cover’ the whole region that they are located in. Others, however, 
including CLEAR Secretariat staff, noted that the original intent was merely to encourage centres to work 
across borders, but without expectations for how many countries this should entail. 
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continue, CERP was not able to do that, resulting in almost no activities being conducted in 
Pakistan during this time. 


The delay was due to the government of India’s refusal to release the funds and to discussions 
between the World Bank and J-PAL/CERP on how funds to Pakistan should be reported upon. 
These issues have since been resolved and are not expected to pose further challenges. 


Viability 

In the context of the CLEAR centres, we understand the notion of regional centre viability to 
encompass the following dimensions: i) financial viability, i.e. access to sufficient sources of 
funding to allow the centre to continue its current and planned activities and services; ii) 
institutional viability, i.e. continued ownership of and support from the centre's host institution as 


the regional anchor of the CLEAR centre; and iii) contextual viability, i.e. continued relevance of 
and demand for the services of the CLEAR centre. 1? 


Finding 53: There is a strong likelihood that the existence and work of the CLEAR SA 
centre will be sustained after CLEAR funding ends. 


The optimistic assessment of the likely viability of the SA centre is based on the following factors. 


Planning for financial viability: The CLEAR SA team is dedicated to the purpose and work of the 
centre and has engaged in sustainability planning. The centre established an endowment fund with 
approval from the CLEAR Secretariat. Revenues generated from paying clients are paid into this 
fund, thereby contributing to establishing a financial foundation for its sustainability. 124 


Integration in and support from host institution(s):CLEAR’s objectives and mandate are closely 
aligned with and relevant to the mandates and activities of J-PAL SA, IFMR, and CERP, and all 
three organisations view the CLEAR initiative as a valuable complement to their regular work. J- 
PAL South Asia has committed to continue funding key professional staff positions within CLEAR 
and to provide administrative and finance staff. Similarly, J-PAL SA and IFMR have indicated that 
the centre will continue to have access to their respective networks of associates. 


Context: Document review and stakeholder consultations indicated not only that there is 
considerable demand for M&E capacity building and advisory services in India (and the region), but 
also that there are a number of potential sources of funding that the CLEAR SA centre is, or is 
planning on, tapping into. The centre’s draft business plan (2013) outlines a variety of strategies for 
both cost-recovery*” and profit generation’ that it envisages to apply. The centre also intends to 
engage in fundraising by targeting donors (building on existing funding relationships with DFID, 
USAID and 3ie) and, possibly, private sector companies. Additional opportunities may exist 
through collaboration with actors such as UNICEF, UN Women, and the World Bank who have an 
interest in strengthening M&E capacity especially at sub-national levels, but who do not have the 
internal capacity to develop and implement related measures. 


3.4.5 Forward looking considerations 


The following suggestions for consideration by the CLEAR SA centre derive from the findings 
outlined above. 


e As the CLEAR SA centre develops its strategy for the next years, it should capture key 
assumptions about the envisaged change processes in a visual representation of its 


123 This dimension relates to the ability and willingness of national/regional actors to pay for CLEAR services. 
124 At the time of writing no information was available on the amount currently in this fund. 
125 Payment to centre for training and advisory, direct logistical assistance, co-hosting and co-delivery. 


126 Paid open enrolment courses, paid advisory services and custom made courses, and e-learning courses. 
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internal Theory of Change, and clearly link this ToC to the internal results framework 
that it is currently developing. '?7 


We support the RAC suggestion that the SA centre may want to frame its work in terms 
of one or more research questions. This could help the centre further focus its work on 
a number of clearly defined roles and/or areas of engagement and also make valuable 
contributions to the creation of global knowledge. The research expertise and 
experience of both SA centre and J-PAL SA staff are likely to be highly valuable in this 
regard. 


Further elaborating on the envisaged use of demonstration projects or pilots at different 
levels could help the SA centre clarify its strategy and its approach to various donors. 
This should include a mapping of key actors in India and/or the region in terms of their 
potential influence and to inform the centre’s engagement with specific clients or 
partners on national, sub-national, or sectoral M&E systems. 


The SA centre’s development of an internal monitoring plan linked to its results 
framework should help the centre capture longer term changes to which it contributes. 
Ideally, this plan should also include ideas (and proxy indicators) for how to capture the 
often evasive results of advocacy work and other efforts aimed at influencing demand 
for well-implemented M&E, and eventually demand for and use of evidence generated 
by M&E. 


We commend the SA centre on its intent to develop a Regional Strategy, and suggest that this 
strategy should outline not only envisaged types of activities, but also the specific results (at 
national or regional levels) that its regional engagement is envisaged to contribute to, and/or what 
types of learning it will inform. 


127 The J-PAL proposal to host the CLEAR SA centre (2010) included a draft ToC that was in many ways 
more elaborate than the current overall CLEAR ToC. This may provide a good basis to start from should the 
centre wish to develop an updated version based on its experience and evolved thinking. 
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3.5 CLEAR East Asia (EA) Centre 
3.5.1 EA Centre Profile 


Selection and establishment 


The CLEAR East Asia centre — based at the Asia Pacific Finance and Development Centre 
(AFDC) in China based at the Shanghai National Accounting Institute (SNAI) - is the only of the 
currently existing centres whose host institution was not selected via a competitive process. Before 
CLEAR was developed, the World Bank, Asian Development Bank, and other partners in 2007 
came together to launch the 


Shanghai International Program in SHIPDET was jointly launched by the Chinese Ministry of Finance 


Development Evaluation Training (MOF), the World Bank Group, the Asian Development Bank (ADB) 
(SHIPDET). SHIPDET has enjoyed | and AFDC. Each year, SHIPDET delivers two training workshops in 
success in training evaluation spring and fall respectively. The spring workshop is designed for 
practitioners, disseminating Chinese participants, primarily governmental officials. The fall 
evaluation knowledge, and workshop is designed for international participants, primarily officials 


networking, not only in China, but responsible for development evaluation from governmental 

Iso in th hol : When th departments of Asian developing countries and representatives of 
aso 1N INe wie region. ening bilateral and multilateral development agencies. SHIPDET follows 
CLEAR program was begun to the World Bank IPDET core courses. 


pe allan was Chinese participants for SHIPDET training are selected and 

y financed by the Chinese government as the MOF is promoting 
selected to host the EA centre performance evaluation. International participants are mainly 
based on its existing relationship selected and supported by ADB. 
with the World Bank around hosting 
and facilitating SHIPDET since 2007. 


In July 2010, a Memorandum of Understanding (MOU) for the establishment of the EA centre was 
signed between IEG, the Chinese Ministry of Finance (MoF) and AFDC. In October 2012, AFDC 
received a grant of USD 350,000 from CLEAR, which marked the official start of the EA centre. 


Financial resources 


At the end of December 2013, the Chinese MoF requested CLEAR to extend the closing date of 
the grant from 31 December 2013 to 31 December 2014. The principal reason for this request was 
that “although several activities were implemented as planned, the budget was underutilized due to 
co-financing agreements, leaving a surplus for the year. As of December 2013, approximately 
USD 151,961 of the total grant funds of USD 350,000 were spent and USD 198,039 remained 
unspent.” The request was approved by CLEAR. 


Staffing 

At the time of the evaluation site visit (March 2014), a 
the EA centre staff comprised two full-time staff and Type of activity a ae e 
four part-time staff from AFDC. The CLEAR team is SS 


led by a director. Daily operation is coordinated by two 





full-time programme officers with the support of two LÉ a 
junior level staff. According to AFDC, other AFDC staff Knowledge exchange seminars 3 
members also provide support for CLEAR activities 
during the hectic training season. Technical assistance/advice 8 
Centre activities 

Knowledge resource 1 


To date, the EA centre has offered two kinds of 

regular training programmes: SHIPDET and training on Impact Evaluation. In addition, it has 
offered ad hoc training on Performance Budgeting. As shown in the sidebar, the centre has 
engaged in a number of advisory services, knowledge exchange activities, as well as internal 
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capacity building efforts for its staff. To date, it has developed one knowledge resource. In addition 
to the core course deliveries, like IPDET and Impact Evaluation, the SHIPDET program also has 
specialized courses (usually of two to three days) on a range of topics. Recent specialized topics 
include Performance Based Budgeting, Introduction to Impact Evaluation, Cost-Benefit Analysis for 
Evaluators, and the Case Study Approach to Evaluation. 


The centre does not have confirmed data on the composition of its clients, but estimates that 50% 
of clients are government officials from different countries, 20% are from academia, while the 
remaining 30% are evaluation practitioners, including independent consultants, as well as staff 
from auditing and evaluating consulting firms. 


3.5.2 EA Centre Context 


Finding 54: The establishment of the EA centre took place in a regional environment 
characterised by national governments (in particular China) increasingly 
pushing for and establishing systems for M&E and performance-based 
budgeting. The supply of quality RBM/M&E services was scarce, and 
awareness of the benefits of high-quality M&E and RBM not yet wide spread or 
institutionalised. This context has not significantly changed. 


Given the location and nature of its host institution as an entity sponsored by the Chinese 
government, the EA centre has been primarily focused on addressing gaps and needs in the 
national context of China. This is reflected in the following observations on the regional context. 


e Across countries in the region, and in China in particular, national governments had put 
increasing emphasis on M&E and performance/results-based management and 
budgeting, and had taken steps such as establishing dedicated M&E units. In China, 
the government had conducted several M&E pilots and had made efforts to promote 
RBM practices in all line ministries and at all levels of government. Since 2011, all of 
central government-funded projects of over 5 million RMB (amounting to hundreds of 
billions RMB per fiscal year) must be monitored and evaluated by budgeting 
departments (i.e. by evaluation professionals or government officials). However, staff 
members often lacked the knowledge and skills to put these requirements into practice. 


e M&E capacity building opportunities in the region were scarce and concentrated on 
internal government training. The creation of SHIPDET in 2007 by the Chinese Ministry 
of Finance, in collaboration with the World Bank and the Asian Development Bank, was 
a first attempt to address gaps in capacity building opportunities and meet demand for 
M&E services. Building on the momentum and platform created by SHIPDET, CLEAR 
introduces other M&E training topics to the region, such as systematic training on 
impact evaluation. 


e While demand for M&E existed in principle, it was not consistently based on a clear 
understanding of the specific benefits that could derive from rigorous monitoring and/or 
evaluation. The absence of a culture of public constructive criticism (e.g. in China) 
posed an additional challenge to building demand for M&E services and related 
capacity building. Across countries, and particularly in China, there had been no 
tradition of independent evaluators speaking “truth to power”. 


e In pursuing various development reform initiatives, many East Asian countries (e.g. 
China and South Korea) use top-down, government-led strategies rather than bottom- 
up models. Compared with other regions, evaluation associations initiated by civil 
society organisations are less active in East Asia. While they are not well-organised as 
a community, these associations are a growing force calling for accountability in 
disbursing public funds and for better M&E services. 
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While this context has not changed significantly since the EA centre was established, consulted 
centre staff noted that they had observed encouraging developments, especially as regards the 
awareness of and interest in evaluation among various stakeholders, including government 
Officials. 
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3.5.3 EA Centre Strategy and Capacity 
Strategy 


Finding 55: Following the SHIPDET tradition, the EA centre’s core strategy has been to 
strengthen government capacity for RBM and M&E by trying to influence a 
critical mass of individuals through training and knowledge sharing activities. 


According to the CLEAR East Asia Project Document (November 2012-December 2013), the 
centre’s higher level objective is to contribute to helping governments develop and implement 
evidence-based effective development policies and programmes. Accordingly, the centre primarily 
targets government officials, but is also aiming to work with other groups in positions and/or with 
skills likely to drive M&E practice and culture in China (e.g. post-graduate students and chartered 
accountants). 


The centre strategy, as outlined in its project document, encompassed three pillars: 
1. Provide training to relevant evaluation practitioners and managers or decision makers 


2. Act as an evaluation knowledge supplier, e.g. by conducting research on international best 
practice relevant to specific applications in the Chinese context 


3. Become a leading evaluation practitioner's network, e.g. by promoting evaluation 
knowledge through its website, conducting case study seminars, and publishing 
newsletters, thereby inviting participation of and contributions from evaluation practitioners. 


One additional feature to the CLEAR EA strategy that has emerged during implementation is to 
complement these pillars with demand-driven technical assistance and advisory services to 
individual clients. 


Centre activities to date have been strongly driven by requests from the Chinese government, and 
imply the underlying assumption that reaching a critical mass of staff in an organisation or system 
is likely to contribute to changes in individual and collective behaviour. '7° 


Document review and stakeholder consultations did not provide additional information on the 
underlying centre-specific theory of change, e.g. as regards deliberations about the respective 
roles of actors at national and sub-national levels, or the envisaged transition from strengthening 
the awareness, knowledge and skills of individuals to influencing organisations and eventually, 
systems. 


Human Resources 


Finding 56: The EA centre, through its host institution AFDC, possesses strong managerial 
capacity. The centre’s current professional capacity is weak and the delivery of 
capacity building services is reliant on a network of international resources. 


Managerial capacity: The EA centre has had strong managerial and administrative capacity, largely 
due to the fact that it has been able to rely on experienced AFDC staff, including senior AFDC 
leadership. While consulted AFDC and CLEAR staff noted that the administration of CLEAR (e.g. 
reporting obligations and grant management) had at times been cumbersome, they had no 
significant difficulties carrying out related tasks. 


128 This is also an implicit assumption underlying the SHIPDET model. 
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Professional capacity: The current CLEAR team includes two full-time staff and two part-time staff 
under the direction of two senior staff. Most staff members focus on administrative tasks.!?* For the 
delivery of capacity development activities, the EA centre has used the services of a few senior 
AFDC staff and various international experts. The international SHIPDET training course and the 
Impact Evaluation training module have been delivered by international experts. While the Chinese 
session of SHIPDET is led by AFDC staff members, these, as well as consulted international 
instructors, have expressed the need to ensure the continued quality of this module, e.g. by 
building additional domestic capacity to teach the course in future, or by engaging Chinese- 
speaking overseas experts who understand the Chinese context. 13° 


Another challenge is that the centre has very limited access to additional (or substitute) resources, 
in the event case that a scheduled instructor cannot deliver due to health or other reasons. This is 
partly due to the fact that AFDC staff members do not necessarily have the knowledge, skills and 
experience required to teach CLEAR courses, be it SHIPDET or training on impact evaluation. It 
also reflects the continued dearth of skilled instructors based in China and the wider region who 
possess sufficient knowledge and experience to be able to provide high quality capacity building on 
M&E and performance management. 


Inter-institutional linkages 


Finding 57: The EA centre is building strong partnerships with the government of China 
and with public finance institutions in the region. It has not proactively reached 
out to other development actors, including multilateral and bilateral agencies, 
CSOs and academia. 


To date, inter-institutional linkages that the EA centre has established have focused on actual and 
potential government clients, primarily in China (e.g. Guangdong province, Shanghai Financial 
bureau). Being directly under the administration of Chinese MoF, AFDC has been able to use the 
MoF network to reach out to other ministries of finance in neighbouring countries. For example, it 
provided a series of workshop on budget performance evaluation for the Ministry of Economy and 
Finance of Cambodia. Further expanding this type of client base in other countries in the region is 
a strategic priority of the centre. 


The EA centre has also built long-term partnerships with development agencies that either provide 
funding for CLEAR EA activities or sponsor participants to take part in CLEAR training workshops. 
These include the Asian Development Bank (ADB), the World Bank (WB), and the International 
Fund for Agricultural Development (IFAD). However, the EA centre’s interaction with other 
multilateral and bilateral agencies has been limited. Consulted representatives of such other 
agencies noted that they have been trying to identify nearby resources to help strengthen the M&E 
capacity of their own offices and partners, but were unaware of the CLEAR EA centre’s existence. 


To develop a network of evaluation practitioners, the centre staff has cooperated with the Shanghai 
National Accounting Institute (SNAI) to provide training courses on M&E and Performance Based 
Budgeting for Chinese certified accountants. On the other hand, the centre’s interaction with CSOs 
and academia has been limited. In China, several CSOs are providing M&E training courses and 
some scholars are providing evaluation services to government agencies. To date there has not 
been any coordination between these actors and the EA centre, nor have they explored 
opportunities for synergies or collaboration. 


129 Consulted staff members expressed a strong interest in eventually broadening their involvement beyond 
administrative tasks and get more involved in the delivery of capacity building interventions, e.g. as teaching 
assistants. 


130 The 2013 Annual Report notes that within the next five years, CLEAR-East Asia aims to conduct about 
50% of all lectures, technical assistance and professional guidance with its own professional staff. 
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Due to China’s strong economic growth, development evaluation training offered in Shanghai has 
attracted some participants from outside of the region (e.g. Africa, Central and South Asia). Due to 
this, the EA centre has advocated that the CLEAR programme consider removing the geographic 
boundaries between the regional centres and allow applicants to choose where they receive the 
training. In addition, the EA centre has also called for increasing bi/trilateral exchange and 
cooperation among the regional centres by secondment arrangements or short-term staff 
exchange. 


Host Institution Arrangement and Governance 


Finding 58: Overall, the host institution arrangement with AFDC has worked well for the 
East Asia centre, especially in view of providing it with credibility and contacts 
in the Chinese context. 


Although directly reporting to the Chinese MOF, AFDC has full autonomy in administering and 
managing the CLEAR programme. AFDC has repeatedly and explicitly expressed its interest in 
and commitment to continuing and building on its SHIPDET experience; senior leadership 
(including the head of AFDC) have taken an active interest in the CLEAR centre’s work. All current 
CLEAR staff members are also AFDC employees, which has contributed to the integration of the 
centre into its host institution. At the same time, senior AFDC staff members tend to be busy, and 
have not always been able to dedicate their full attention to the centre. 


AFDC’s affiliation with the Chinese Ministry of Finance provides the CLEAR centre with the 
required legitimacy to engage with various units in the Chinese government. Similarly, AFDC’s 
reputation as the host of SHIPDET positively influenced the centre’s ability to reach out to potential 
clients and donors inside and outside of China.'% On the other hand, consulted representatives of 
domestic CSOs and international organizations working in China noted that in their view this 
affiliation has also been a limitation in terms of the EA centre’s ability and vision for reaching out to 
non-government actors. While this view was not shared by CLEAR EA staff, there is no evidence 
yet that the centre has effectively reached out to or established partnerships with non-government 
actors (other than international development organizations) inside and outside of China. 


Finding 59: The EA centre has established a Regional Advisory Committee, but it is not yet 
operational. This has diminished the centre’s ability to validate programming 
plans and decisions through a broader stakeholder group. 


According to its project document, the AFDC is committed to establishing a Regional Advisory 
Committee (RAC) for the EA centre. At the time of the evaluation visit (March 2014), a three 
member RAC had been formed but had not yet met, and the EA centre has not yet developed 
practical guidance on how RAC members are expected to collaborate and coordinate their 
contributions. 


The three selected RAC members are the Chinese MOF official in charge of the CLEAR 
programme; the deputy director of the International Poverty Reduction Centre in China,'*? and a 
former senior ADB evaluation expert. The composition of the RAC reflects the EA centre’s current 
client and partner base and its focus on China. Consulted RAC members were reserved about how 
much effective time they would be able to devote to the centre given their professional and 
personal schedules. 


131 Stakeholder consultations indicated that in East Asia, “SHIPDET” is still a better known brand than 
“CLEAR”. 


132 The International Poverty Reduction Centre China (IPRCC) was jointly established by UNDP and the 
Chinese government. 
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3.5.4 EA Centre Performance 
Relevance 


Finding 60: EA centre activities have been relevant to the expressed needs of the Chinese 
government and to noted regional gaps in the supply and demand for high- 
quality M&E/RBM services in China and the region, and have been aligned with 
the centre’s envisaged strategy. 


EA centre activities have been aligned with its articulated three-fold strategy (providing training, 
linking local with global knowledge and experience, and providing M&E community with inputs). 
They have also been relevant in light of the noted gaps in the Chinese and regional context. 
Activities made sense in view of the centre’s implicit assumption of change being best facilitated by 
creating a critical mass of individuals with increased awareness, knowledge and skills in M&E/RBM 
in different institutions. Key strengths in this regard are outlined below. 


e Critical mass: Through the SHIPDET training, the EA centre has been able to reach 
almost all MoF staff working in the divisions dealing with international aid. In addition, 
the MoF has issued new regulations on performance evaluation on International 
Financial Institutions (IFI) projects to guide these staff to conduct daily evaluation 
activities in line with international good practice. The EA centre is now reaching out to 
other MoF staff and public servants in other departments. 


e Supply: Targeting chartered accountants as a group of actors likely to be able to 
acquire and use M&E skills in their daily practice was a reasonable, pragmatic 
approach to addressing the noted absence of appropriate M&E capacity, and the 
urgency of the government's need to address related gaps. 


e Demand: Given that many of the centre’s activities have included both mid-level officers 
and higher ranking officials and policy-makers, the centre hopes to influence not only 
the supply of relevant M&E/RBM expertise, but also demand for such services by 
raising awareness among relevant actors of the potential benefits. 


Effectiveness 


Finding 61: At midterm, the EA centre has contributed to strengthening the awareness, 
knowledge and skills in M&E and performance management of a considerable 
number of individuals. Achievements to date have the potential to contribute to 
positive changes within government agencies (primarily in China). The centre 
does not yet have adequate systems to capture the longer-term effects of its 
work. 


After two years of operations, the EA centre has reached a considerable number of individuals, 
primarily staff within the Chinese MoF, but also representatives of other government units and 
organisations from China and other countries. 


To date, evidence of actual results achieved through the centre’s capacity building activities is 
largely anecdotal and derived from stakeholder consultations. Interviewed CLEAR clients noted 
that CLEAR EA training had changed their own and — based on their observations — some of their 
colleagues’ or supervisors’ understanding of management, resulting in a shift from focusing on 
inputs to focusing on outcomes (results). Some individuals also noted that the training had 
contributed to changing their understanding of the nature and purpose of monitoring and 
evaluation, which they now understood not as a mere mechanism for control, criticism or 
punishment, but as a potential tool for learning. However, consulted training participants did not 
comment on whether or how their changed perceptions had — until now - resulted in changes in 
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their own practice or that of their organisation, e.g. in collecting, analysing and presenting data or 
applying recently gained M&E knowledge to policy formulation. 


The evaluation found a handful of instances pointing towards increased demand for M&E capacity 
building deriving from the centre’s activities. For example, the city of Shanghai invited an expert 
from the centre to advise them on designing a municipal M&E system. Also, the Chinese MoF 
invited the centre to organise a seminar on best practices in M&E for provincial officials from 
financial bureaus to raise awareness of evaluation to support government plans to roll out a larger 
government-wide M&E plan. 


Furthermore, there is one example of CLEAR’s work having contributed to positively influencing 
the enabling environment for M&E in China. Consulted Chinese government officials 
acknowledged that CLEAR training had helped them conduct research into existing differences 
between international good M&E practice (i.e. IFl's evaluation systems) and domestic practice (i.e. 
China's fiscal evaluation system). Research findings were used by the MoF in developing and 
issuing new Regulations on 
Performance Evaluation on IFI 
projects in China. 


Follow up 
The EA centre does not yet have SHIPDET instructors have developed a listserv for English-speaking 
sufficient systems in place that participants that provides three functions for SHIPDET alumni: (1) a 
would allow it to capture the job centre that provides information on job openings, (2) a vehicle to 
emerging longer-term effects of its disseminate publications on M&E, and (3) a technical assistance 


work on individuals, organisations centre for those who need help from their peers. 


or M&E systems. Some information 

is being generated through regular events organised by the centre for SHIPDET alumni that allow 
them to share their experiences with applying the knowledge and skills gained through the 
SHIPDET training. See also sidebar. However, the centre has not yet established other means to 
follow and capture evolving changes in the different units that it has worked with. Thus, there is no 
strong data yet on the centre’s actual or likely influence (as opposed to its reach). 


Finding 62: The EA centre’s activities have been attended by actors from various countries 
in and beyond the East Asia and Pacific region. However, the centre has not yet 
defined the contributions it hopes to and is able to make outside of China. 


Training activities conducted by the EA centre have included government officials from the Greater 
Mekong Region (such Lao, Vietnam, Cambodia), from central Asian countries (such as Tajikistan, 
Uzbekistan, Kazakhstan, and the Kyrgyz Republic), as well as from the Pacific (such as Samoa 
and Fiji). The centre also conducted a tailor-made training for government representatives from 
Cambodia. In this sense, the CLEAR EA centre has been ‘regional’ in that many of its services are 
open to participants from other countries — often sponsored by actors such as the ADB. The strong 
interest in SHIPDET from actors in these countries likely reflects the noted scarcity of similar 
capacity building opportunities in the region. 


Nevertheless, the centre’s main focus has been on addressing the needs of the government of 
China.'*? It has not conducted any formal or informal needs assessments in other countries, nor 
has it developed an implicit or explicit regional vision and action plan that identifies the strategies 
for, or envisaged results of, engaging with stakeholders from other countries. While the centre’s 
approach and intent to help create a critical mass of trained individuals within targeted 
organisations in China is clear, this is not the case for its engagement with actors from other 
countries. 


133 Consulted CLEAR EA staff, clients and partners widely agreed that addressing the needs of the Chinese 
context alone constitutes an enormous task. 
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Efficiency 


Finding 63: To date, the majority of CLEAR EA centre activities have been funded by other 
sources than its modest sized CLEAR grant. 


By the end of FY2013, the amount expensed or committed by CLEAR to the East Asia centre 
(including grants and direct support from the Secretariat) was USD 422,974, the lowest among the 
existing CLEAR regional centres.'** The same applies to the total projected amount of 

USD 922,974. 


Resources from the CLEAR grant have been used almost exclusively for client capacity building 
activities, with only a small amount being used for internal capacity building of EA centre staff. The 
majority of activities conducted by the EA centre to date have, however, been paid for through 
other sources. '*° The Chinese MoF has been supporting AFDC and thereby also CLEAR operation 
costs (mainly staff salaries) and has been sponsoring the participation of Chinese participants in 
various CLEAR training courses. So far, the Chinese government has contributed at least USD 3.5 
million to the CLEAR EA programme. 


Consulted EA centre staff expressed pride in having been able to engage various domestic and 
international resources, which — in case of the Chinese government — can also be seen as an 
indication of national ownership and political support for the work of the centre. Centre staff also 
raised concerns, however, over how the EA centre's performance compared to that of other 
CLEAR centres. Given that the EA centre has received considerably less resources from CLEAR 
than other centres, the EA team felt that it was neither realistic nor fair to expect it to reach the 
same objectives as the other centres (or to make progress at the same pace). Team members 
noted that in their perception CLEAR reporting guidelines put pressure on them to do so — e.g. as 
regards the expectation for the centre to be diversifying its topics and methods, and reaching out to 
the wider region. 


Viability 


Finding 64: While it is very likely that AFDC will continue to provide and facilitate M&E 
capacity building interventions, the incentives offered through the CLEAR 
initiative may not suffice for the institution to continue hosting a CLEAR centre 
in the longer term. 


This assessment is based on the following observations. 


Sustainability planning: The East Asia centre, and its host institution AFDC, have not yet 
developed a long-term strategic or business plan for whether or how they will approach the 
financial viability and continued existence of the centre. Consulted centre and AFDC staff largely 
attributed this to the very limited size of the CLEAR grant, which, in their view, had not permitted 
them to engage in longer term planning. 


Integration in and support from host institution: While AFDC has provided ongoing managerial, 
administrative, and professional support to centre activities, it has made limited investments in 
CLEAR in terms of allocating senior and full time staff resources. This limits the centre's capacity to 
engage in strategic (financial and programmatic) longer-term planning. Also, the question arises 
whether the co-branding arrangement of running SHIPDET under the CLEAR umbrella provides 
sufficient incentives for AFDC to continue with it , or whether it might be more beneficial for AFDC 


134 Relative allocations of CLEAR funds to the different centres reflect CLEAR donor priorities, which are 
linked to assessments of the relative need for development assistance in different regions. 


135 For example, during the period Q1&2, 2014, the centre’s total budget was USD 356,958. Of this, 
USD 44,027 came from the CLEAR grant, while the rest came from other sources, in particular funding from 
the ADB. 
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in terms of administrative and reporting requirements to go back to hosting SHIPDET as a stand- 
alone program. 


Context: Document review and stakeholder consultations indicate that there is ample demand for 
M&E capacity building services in China as well as in the larger East Asia region. The Chinese 
MoF will almost certainly continue its support for the operation costs of AFDC, as well as its 
sponsorship of Chinese participants for training courses such as SHIPDET, regardless of whether 
these are provided through AFDC or CLEAR. While it is unsure whether the Chinese government 
is likely to be interested in investing in the expansion of the EA centre’s current work (e.g. 
deepening its regional approach, or adding a more distinct research component), other donors with 
an interest in the East Asia/Pacific region (e.g. the ADB or AuSAID) may be willing to explore 
related options. 


Overall, there are a number of positive factors that can support the financial viability and continued 
relevance of the CLEAR EA centre (in particular the demand for M&E capacity building and the 
availability of likely sources of funding). The bigger question, however, is whether the benefits of 
hosting a CLEAR centre (in terms of financial resources, contacts/networking opportunities, 
reputational affiliations) outweigh the drawbacks for AFDC (e.g. added reporting obligations and 
administration). 


3.5.5 Forward looking considerations 


1. AFDC has to decide whether (or under what conditions) hosting CLEAR continues to make 
sense for AFDC. 


2. If AFDC is committed to continue hosting CLEAR EA, the centre needs to develop a clear 
vision and strategy for its engagement both in China and the broader region. This should 
clarify whether and to what extent the centre’s work will continue to focus on China or be 
aimed at the broader region, and with what envisaged results. 


3. In this vein, developing a centre-specific theory of change could be helpful in making 
explicit what “success” is envisaged to look like (at national or regional levels), and how the 
centre sees itself contributing to this success. 


4. There are a number of areas where the EA centre could act as an important player and to 
contribute to regional capacity on M&E and RBM. For example, the centre aspires to 
become a regional knowledge hub on M&E. However, this would requires it to be more 
proactive in reaching out to a broader and more diverse community of evaluation practice, 
particularly the donor community and the CSO community that have already developed 
good practice and lessons learned in their own M&E domains. 


5. Closely linked to its intended future vision and strategy, the EA centre needs to identify its 
capacity needs and how it envisages to meet these needs through either internal or 
external resources and/or capacity development. 


6. As soon as possible, the EA centre, in consultation with the selected RAC members, should 
determine realistic and useful ways for the RAC to meet (either in person or virtually) and 
provide input to the centre's ongoing work and strategic planning. Depending on the 
centre's future foci it, together with the RAC, may consider further diversifying RAC 
membership (e.g. by including a representative from a non-governmental organisation 
and/or another country in the region). 


The centre should explore additional ways of how to systematically monitor and capture emerging 
insights on longer term contributions that its capacity development work is having on individuals, 
organisations, and eventually on M&E systems. 
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3.6 Francophone Africa (FA) Centre 
3.6.1 FA Centre profile 


Selection and establishment 


In October 2011 the Centre Africain d'Etudes Supérieures en Gestion (CESAG) in Senegal was 
selected from 24 applicants from 11 countries to serve as the CLEAR host in Francophone Africa 
(FA). In its proposal to host CLEAR, CESAG expressed the intention to partner with the 
International Institute for Water and Environmental Engineering (2ie), in Burkina Faso as its affiliate 
centre. 


Financial resources 


CESAG and the World Bank signed a grant agreement on 24 October 2012 for USD 268,970. In 
addition to the CLEAR grant, the International Development Fund managed by the World Bank is 
contributing approximately USD1 million of support to CESAG for the CLEAR centre over the three 
years of 2013, 2014 and until November 10, 2015. 


The FA centre received and began to draw down the first tranche of funding from the CLEAR grant 
in February 2013. According to CLEAR annual reports, up to July 2013 the centre expended USD 
50,192 from its CLEAR grant. 


Initially the centre was operating with one full-time person (the coordinator), one part-time training 
specialist, one financial officer who later left the centre due to unsatisfactory performance, anda 
part-time procurement officer. An intern officially joined the team in February 2014 and a secretary 
in February 2014. In early March 2014 the coordinator moved into a new role as Director of the 
department of research and consultancies, a newly created department, as part of CESAG’s 
transformation to business school and a new coordinator was appointed. The coordinator of the 
centre now reports directly to the Director-General of CESAG whereas his predecessor reported 
through the Secretary General. Consulted centre staff noted that, in their view, this direct access 
improved lines of communication and reduced paperwork (although this remains onerous). 
Currently, there is a financial specialist (who oversees all the donor-funded programs within 
CESAG) who partially works for CLEAR with support of his assistant. A part-time procurement 
consultant has been hired since May 2014. 


Centre activities 


By the end of 2013, the FA centre 

had conducted or been involved in In 2013, following a thorough needs assessment, the FA centre 

23 capacity development developed modules for short-term training in monitoring and | 

activities, 13 including seven training evaluation. These were used in two sessions for 47 trainers held in 
Dakar and Ouagadougou. The centre also organised two regional 

ee pe nee exchange training seminars in Dakar for M&E professionals in 2013. 


assistance/organisational capacity building, one diagnostic, one knowledge resource, one 
assessment/project advisory, and one intervention related to supporting professional 
networks/communities of practice. The centre routinely invited members of CESAG and its staff to 
join events/activities it was supporting (e.g. the M&E for professionals workshop which many 
members of the CLEAR CESAG team and the faculty of CESAG attended). Centre clients have 
included representatives from government agencies, non-governmental organisations, and 
academia. 


136 The CLEAR Annual Report noted 18 activities by the end of June 2013. The remainder have occurred 
since. 





e-Pact 82 


CLEAR Midterm Evaluation 





3.6.2 FA Centre Context 


Finding 65: The FA centre was established in a regional context characterised by growing 
demand for M&E capacity building among government and non-government 
actors, as well as by very limited supply of high-quality related services, and 
low level of existing M&E/PM-related skills and experience among national 
actors. In addition, the region was, and continues to be, one of the most 
poverty-stricken in the world, with low overall education and literacy rates. This 
challenging context has changed only modestly since centre establishment. 


Key characteristics of the regional context in Francophone West Africa at the time of centre 
establishment as regards the existing supply of and demand for M&E services and related capacity 
building are outlined below. 


e Since approximately the beginning of the new millennium, a growing number of national 
governments have acknowledged the need for introducing results-based management 
approaches and performance evaluation. Some governments embarked on reforms of 
their systems for planning, programming, and budgeting, but efforts were challenging 
due to limited resources for M&E combined with scarce capacity in the field and the 
unstable nature of the region as a whole. 


e Nevertheless, in some countries (e.g. Benin, Burkina Faso, Mali, and Senegal) M&E 
had been identified as a national priority. As a result, the respective governments had 
expressed considerable need for awareness raising and capacity building among policy 
makers and other actors from the grass roots level to all levels of government. Demand 
for M&E capacity building was also expressed by civil society organisations to help 
them fulfil their intended functions as watchdogs, drivers of government reforms, and/or 
service-providers (i.e. based on evidence to inform advocacy and planning). 


e There was a growing evaluation community in the region, but individual members often 
lacked formal training. In addition, evaluation was not widely recognised as a 
profession, and most active evaluators did not hold a certificate or diploma. 
Furthermore, programmes leading to M&E certificates or diplomas tended to lack depth 
and practice-orientation. The fees charged by capacity development providers were 
high by local standards — in the range of USD 3000 - and few students were self- 
financing. 

e Inthe last decades, countries of the sub-region have started developing evaluation 
associations — Niger (2001), Benin (2008), and more recently Senegal in 2010. SenEval 
members include several experienced evaluators, some of whom also have training 
backgrounds, such as the current coordinator of the CLEAR FA centre. There is 
potential for off- loading some of the routine M&E training onto either SenEval or some 
of the NGOs working in the field who have practical in-the-field experience. 


e CESAG is in the process of converting itself into a business school. The DG sees the 
potential to use CLEAR as a means of strengthening his institution as a whole. Getting 
the CLEAR Centre running smoothly is thus critical to CESAG’s reputation and its ability 
to access more resources. 


Factors not specific to the issue of M&E capacity, but relevant in view of the broader context in 
which the FA centre is located, include the fact that Francophone Africa is a large and ethnically 
diverse region that includes some of the poorest countries in the world. This has also resulted in 
considerable deficits in education and literacy rates. Travel between countries in the region is 
expensive and time consuming, making regional exchange more challenging than in other 
environments. In addition, many countries in Francophone Africa continue to have weak institutions 
unable to secure their respective national territories, let alone ensure equal treatment of different 
ethnic or religious groups. Civil conflicts have been frequent. In many national contexts, the 





e-Pact 83 


CLEAR Midterm Evaluation 





concept of a national government that is accountable to its citizens was and continues to be not 
closely related to existing reality. 


3.6.3 FA Centre Strategy and Capacity 
Strategy 


Finding 66: The FA centre conducted three comprehensive assessments to explore 
demand for M&E and performance management information and services in 
three countries. The centre is still in the process of translating assessment 
findings into an explicit strategy. This makes it difficult to assess whether and 
how individual capacity development interventions carried out to date fit into a 
broader vision for change. 


In February 2012, the FA centre, with support from the CLEAR Secretariat, launched assessments 
of the context, need, and demand for M&E in Benin, Mauritania, and Senegal to inform its strategy 
and ensure relevance of centre’s programmes. The studies were completed in early 2013 and are 
available on the global CLEAR website. Key findings included that the three countries were at 
different stages of development of their M&E systems and required different entry points for 
capacity building. At the time of the evaluation site visit (February 2014), the centre had not yet 
translated these findings into an explicit strategy (or country-specific strategies).*?” This delay has 
been due to the fact that the centre coordinator has been occupied not only with the 
implementation of activities but also a heavy burden of administrative procedures. The centre has 
since engaged an external consultant who produced a draft strategic plan in May 2014. The 
document is envisaged to be finalised following the CLEAR global forum in September 2014. 


Capacity development activities conducted to date appear to have been guided by the broader 
objective to establish the centre as a provider of customised M&E training and capacity 
development services with a focus on strengthening the supply side of M&E. Key strategies in this 
regard were to: 


e provide a variety of capacity building interventions (in particular training) to a variety of 
government and non-government actors involved in conducting or managing 
evaluations in order to increase their knowledge and skills of M&E in general, as well as 
of more specific areas such as Impact Evaluation 


e help develop the skills of M&E trainers both in Senegal and other countries to increase 
the number of individuals able to provide M&E capacity development services in the 
region. 


In addition, the centre’s project document notes the intent to provide advisory/knowledge service, 
evaluation and applied research. To date this aspect has not yet strongly been reflected in actual 
centre activities, with exception of the demand study. While this study indicated that the centre’s 
work during its first year would place particular emphasis on the three countries covered in the 
study, this has only been visible for Senegal, but not for Benin and Mauritania. 


It is also anticipated that there will be a business plan which will define target groups, themes and 
types of services to be offered. The centre coordinator is then expected to obtain buy-in for this 
and the strategic plan from the CESAG Board. 





137 The overview of the demand studies (page 39) identifies the need for the FA centre to “develop a strategy 
that continues to examine future demand for M&E services in the region, but is also capable of providing 
both the supply and demand side of the services in demand”. The document further identifies centre 
activities for Year One of its operations, focusing on the development of the centre’s internal capacity and 
resources, and on delivering M&E services within the region, with focus on the three countries covered in the 
demand assessment. 
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Human Resources 


Finding 67: The FA centre has recognised strengths in providing M&E training. Its 
managerial experience and its professional capacity in other areas have been 
less strong. However, recent staff changes have brought improved 
management skills and the new coordinator is seeking opportunities to 
strengthen the skills and knowledge of both Centre and CESAG staff. 


The FA centre has highly qualified professors who have a sound knowledge of methodological 
approaches typically used in the evaluation field, although they may not have experience as 
practitioners. As professors, they are competent at teaching and training and they have also 
worked as consultants proving advice to the government reform process. 


At the same time, the CLEAR FA team has faced a number of challenges since centre 
establishment. Team members had limited, pre-existing managerial know-how and were unfamiliar 
with WB rules and regulations. This resulted in centre staff having to spend considerable time and 
energy on learning how to navigate the centre’s administrative environment. Task completion often 
required hands-on support from the CLEAR Secretariat. 


The new coordinator who came on board in March 2014 has a good knowledge of the CESAG 
systems, practical experience with World Bank and donor procedures, and also prepared the 
CESAG business plan. He is therefore well placed to ensure a degree of alignment of CESAG and 
Bank systems and to develop strategic and business plans. Despite this recent addition to the staff, 
the centre still does not have enough experienced staff to meet existing demand for capacity 
building services. The new coordinator is currently trying to provide incentives for CESAG staff to 
develop their own skills and to get more involved in CLEAR activities. 198 


Inter-institutional linkages 


Finding 68: The CLEAR FA centre is still in the process of establishing formal partnerships, 
including with its intended affiliate centre in Burkina Faso. 


The FA centre has developed good working relationships with some prospective clients, especially 
the Senegalese government. At the same time, its formal links with other organisations (be it 
potential clients or partners) are not yet strong. The most advanced partnership negotiations have 
been carried out with the UNICEF Regional Office for Western and Central Africa, but have been 
delayed, with a formal partnership agreement expected to be signed in October 2014. 


Beyond that, the FA centre has not made significant progress in identifying or approaching other 
potential collaborators who would be able to complement the centre’s internal capacity (e.g. other 
training providers who might be able to conduct or take over some of the centre’s envisaged 
regular, basic training programmes).*?** This, together with the noted scarcity of M&E professionals 
with the required expertise and experience, has meant that the centre has had only limited access 
to external professional expertise in the region to complement its internal resources. Consultations 
with CLEAR FA staff and clients also indicated that under its previous coordinator the centre 





138 Some examples of incentives include: Two young CESAG instructors were chosen for training in impact 
evaluation in China and will develop a similar course for CLEAR with guidance from an international 
consultant. Because CLEAR can't use either IDF funds or CLEAR funds to pay CESAG staff for their 
services, the coordinator wants to use any profits generated from training programs for this purpose. There is 
a limit, however, to how much this approach can be used. The Centre’s mandate to produce public goods will 
limit its ability to generate profits and thus how much this mechanism can be used. 

139 The recommendations of the 2013 diagnostic report of CESAG included a) that the FA CLEAR centre 
should establish a lasting partnership with SenEval, in both training and consulting, so as to take part in the 
evolution of the profession and identify experienced M&E consultants; and b) that it should pursue a 
partnership strategy at the regional level to identify institutions that can best support its activities. 
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tended to perceive and approach other M&E capacity building providers as competitors, rather 
than as potential allies. 


The experience of the last two years suggests that the issue of partnership in the FA context is 
more complex than originally envisaged and that expectations for the initial years of the Centre 
were greater than the political context and the staff available would allow. Arranging partnerships 
where both parties feel a high degree of commitment to joint medium to long-term goals is more 
important than setting up more numerous arrangements focused on one or two activities. 2iE in 
Burkina Faso was one of the early supporters of the CESAG proposal and had been suggested as 
its affiliate centre in part because of its distance learning capabilities which were hoped to prove 
useful for reaching out to other parts of the Francophone Africa region. To date, 2iE has been 
involved in two training sessions offered by the FA centre but the contacts between the two 
organisations appear to be tapering off in absence of a formalised partnership agreement. 


Host Institutional Arrangements and Governance 


Finding 69: While CESAG has been generally supportive of CLEAR, there is room for 
strengthening the collaboration between the FA centre and its host institution. 


CESAG senior leadership has been and continues to be enthusiastic about the CLEAR concept, 
and the centre has benefited from CESAG’s positive reputation as a successful institution involved 
in educating future policy and decision makers especially in the context of the Senegalese 
government. Almost all people interviewed in Francophone Africa noted that, in their view, CESAG 
was the appropriate organisation to host CLEAR, and that there was no other organisation with 
similar qualifications in the whole of Francophone West Africa. It has educated a large number of 
the policy-oriented government officials in Francophone West Africa and is known as a centre that 
attracts thinkers. 


CESAG has provided administrative and operational support to CLEAR, including funding its staff 
positions and providing interim financing before the grant and IDF funds became available. 
However, the Centre experienced significant challenges and spent a lot of time trying to navigate 
CESAG’s antiquated administrative and financial management system (which focuses on control 
and monitoring inputs rather than results), and applying not only these but World Bank and CESAG 
regulations as well.** Although the centre coordinator reports directly to the Director General of 
CESAG, the administrative burden on centre staff has been heavy. 


CESAG is currently in the process of trying to address key bottlenecks and ease the administrative 
burden for CLEAR. It has established a new unit to facilitate contracts and ensure timely payments 
against them. CLEAR FA centre staff 


appreciated these efforts, but also Envisaged Master level program on M&E at CESAG 


a a ee nee cus The interest of CESAG in hosting the CLEAR centre is related to its 
A ie intent to offer an MA level program in M&E, thereby becoming a 
on ensuring accountability, rather centre of excellence for M&E in the context of higher education. The 
than on facilitating and capturing the program is intended to complement the objectives of the CLEAR 
centre’s achievements. centre with its focus on building M&E capacity of practitioners and 
governments in the region. CESAG expects that CLEAR will play 
To date the FA centre has not yet the major role in launching the M&E master’s program in 2015. The 
significantly benefited from access to downside is that this activity will put much pressure on the limited 
CESAG faculty members, despite the resources of the Centre. While the CESAG Board has approved the 
fact that the former CLEAR director Masters program, CESAG has not yet taken concrete steps to put t 
into place." 


140 Several consulted CLEAR and CESAG staff remarked that the practical implications of the CLEAR 
organisational arrangements had not been well thought through at programme onset, and had therefore not 
been taken into consideration when establishing resource allocations. 


141 More progress in this regard is envisaged for 2015. 
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was himself part of the CESAG faculty. As noted above, the new centre coordinator is hoping that 
a new set of incentives will make it more attractive for CESAG staff to engage in CLEAR centre 
activities. See also sidebar. 


Finding 70: The FA centre has not yet established a Regional Advisory Committee. This 
has diminished its ability to discuss and validate its programming plans and 
decisions with a broader stakeholder group 


At the time of the evaluation, the FA centre had not yet established a Regional Advisory Committee 
(RAC). In March 2014, the centre put forward recommendations to the World Bank on who might 
be appropriate members of the RAC, but the Bank asked for more strategic choices and greater 
diversity in terms of gender and regional representation, and in relation to suggested members’ 
experience. The centre is currently (September 2014) working on a revised list. 


The absence of a RAC has limited the FA centre’s ability to have its strategic plans and decisions 
reviewed and validated by a broader group of regional stakeholders. There does not appear to be 
any other permanent forum that CLEAR wants to use for this kind of discussion, although visiting 
groups like the evaluation team provide some ad hoc opportunities. Both the new coordinator and 
the DG of CESAG favour the establishment of a RAC. 


3.6.4 FA Centre Performance 


At the time of this evaluation, the FA centre had been carrying out activities for more than two 
years, but with a full range of services, for only about 18 months. The following findings should be 
considered keeping this in mind. 


Relevance 


Finding 71: The FA centre is relevant to existing demand and needs for M&E and RBM 
capacity building, and to persistent gaps in the supply of related services. It is 
not yet evident, however, what specific development results the centre is 
aiming to contribute to at national or regional levels, and how. 


The overall objective of the CLEAR FA centre (to provide high quality M&E capacity building in the 
region) is relevant in view of the broad existing regional needs and the continued gaps in the 
supply to address these needs. At the same time, centre services have not yet been tailored to 
address the particular needs of specific national (or sub-national) contexts as have been identified 
in the regional needs assessment in three countries. Similarly, the centre has not yet conducted 
similar assessments, nor made explicit its thinking as regards the relevance of its work in other 
contexts, for example (post) conflict settings such as Mali. 


The evident relevance of the centre lies in its intention and potential to fill an existing gap in the 
supply of M&E capacity development, in particular training. The centre has not yet developed an 
explicit (or indicated the existence of an implicit) regional or country-specific Theory of Change that 
would identify one or more development results that its work is aiming to contribute to. In other 
words, it has not yet defined what ‘success’ of its work at national or regional levels will look like. '4? 


As regards the relevance of individual centre activities (e.g. specific training workshops and 
learning materials) consulted CLEAR FA clients provided overall positive feedback, which 
confirmed the positive rating provided for these events in post-training assessments conducted by 
CLEAR FA. The main areas suggested for further improvement of training events were to include 


142 At the time of finalizing this report (September 2014) the evaluation team had not had access to the draft 
FA centre strategy noted above. We are therefore unable to assess whether and to what extent this strategy 
addresses the noted gaps in clarifying the centre’s theory of change and/or expectations of what ‘success’ 
will look like. 
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an even larger number of regionally relevant case studies, and to focus even more on practical 
rather than on theoretical issues. 


Effectiveness 


Finding 72: The FA centre has made some initial contributions to strengthening the M&E- 
related awareness, knowledge and skills of targeted individuals, and to 
broadening the knowledge base on the state of M&E in the region. The centre 
does not yet have systems in place to support and capture the longer term 
effects of its work. 


As is to be expected given the limited time during which the centre has been fully operational, 
evidence of achievements that the centre has contributed to is largely linked to the successful 
completion of specific activities, mostly training, as well as the completion of the regional demand 
studies. Key achievements are outlined below. 


The training sessions facilitated by CLEAR FA have been well received and assessed positively by 
participants (see also sidebar). 
Consulted training participants 


Views on the quality of CLEAR FA trainin 
reported that: eS 2 


Consulted international consultants who have worked with the 


e The events helped raise centre expressed concerns that the training courses they were 
their awareness of M&E involved in had not captured some basic core concepts, that the 
and RBM and their different modules lacked coherence, and that some elements had 
potential benefits in their not been adequately covered (e.g. evaluation standards). 
particular context. While these concerns address the ‘objective’ quality of the events 


(from a developed country point of view), subjective participant 

e They understood the a ratings of the quality of trainings were very high. In fact, several 
value of RBM and its link training participants noted that in their view the centre was setting a 
with evaluation. positive regional standard for high quality work. By this they 

e They felt more motivated referred to the fact that centre activities were well researched and 
and confident to conduct used information and lessons gained from international experience, 
evaluations due to the and that they were delivered in contextually appropriate ways, both 


f in terms of the language of instruction, and in view of the examples 
knowledge and skills used to illustrate issues. 


obtained through the 
training. 

e Participants of the train-the-trainer events gained knowledge, skills, and confidence not 
only in the M&E subject matter, but also in adult education approaches. 


Despite this positive client feedback, one concern of the evaluation team during their visit to 
Senegal was that the FA centre has conducted little, if any, follow up with participants of its training 
workshops, including with the (envisaged future) M&E trainers. This lack of contact was seen as 
constituting a risk that the momentum generated by the training sessions 6 to 8 months earlier 
would be lost, especially as regards the aim of establishing a group (or network) of trainers able to 
independently provide M&E capacity development. The long gap also meant that the centre was 
not systematically collecting information on whether and how participants have been able to apply 
any of the knowledge and skills acquired in CLEAR trainings, and with what effects. # 


The Demand Assessments in Benin, Mauritania and Senegal were the first of their kind conducted 
in the region, and contributed to raising visibility of the centre among stakeholders in the assessed 
countries. The centre has not systematically collected data on whether and how the study has 
been used or by whom. However, several consulted regional stakeholders noted that the study had 
become valued reference material for them. Also, individuals involved in the training of M&E 


143 Since the arrival of the new centre coordinator, there have been several training sessions using the 
services of trainers who had received the ToT Training. 
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trainers noted that this initiative had been informed by the needs identified in the demand 
assessment. The same was noted to apply to the work of the CLEAR FA centre with the 
Senegalese government. 


Finding 73: The FA centre has made efforts to engage actors from countries other than 
Senegal but does not yet have a plan to guide it in defining priorities for and 
clarify envisaged results of its regional engagement. 


Most centre activities carried out in Senegal have involved actors from other countries, and the 
centre has conducted activities in Burkina Faso, Benin, Togo, Mauritania and Niger. However, 
most of the centre’s efforts to engage with actors outside of Senegal have been activity based and 
fragmented. Most of these have been training sessions to which the centre invited organisations 
based on the centre team’s perception that links with these organisations could be useful. 
However, until now the centre’s regional efforts have not been guided by a clearly established 
understanding of the envisaged longer-term objectives of this engagement, and there has been no 
systematic follow-up with the organisations that had sent staff members to CLEAR FA training 
events. 144 


A related challenges is the fact that — as shown through document review and consultations with 
CLEAR FA centre staff - there is uncertainty over how much of the Francophone Africa region the 
centre is expected (or able to) cover. The region is immense, covering 22 countries, with expensive 
and often awkward transportation routes. 


Efficiency 


Finding 74: Challenges in navigating and aligning the administrative environments of 
CESAG and the World Bank have negatively influenced the efficient use of 
resources by the FA centre. 


The process of getting CLEAR FA off the ground was slow. The sudden death of the DG of 
CESAG and a gap of eight months in finding a replacement for him slowed down CESAG 
administrative and financial systems generally and the process of getting CLEAR FA established. 
There was very little activity in CLEAR until February 2013. This resulted in a major gap between 
CESAG’s selection as the host institution in 2011 and the arrival of the first tranche of CLEAR 
funding in February 2013. The CLEAR grant from the World Bank also arrived late because of the 
time consumed in setting up a bank account and other administrative requirements. CESAG did 
provide some interim financing for a few small activities for which it was later reimbursed from 
project funds. The full envisaged centre team strength of six full or part-time staff members was 
only achieved in February 2014. The centre office was equipped with furniture at the same time. 


Centre staff spend a significant amount of time on internal administrative and financial processes. 
This is partly due to the fact that CESAG is a centralised organisation and all financial transactions 
have to go through a number of different vetting steps (up to 15) before being approved by the DG. 
CESAG has recently established a special project monitoring service to speed up the payment of 
accounts and related processes. 


The complexity of regulations around the CLEAR grant has slowed down the payment of accounts. 
(Most consultants interviewed said they had waited six months or more for their accounts to be 
settled.) 


144 According to Secretariat staff working with the FA centre, part of the draft centre strategy developed in 
May 2014 is a 3-step process to engaging with new countries in the region in a more systematic way, 
including: 1) customised training which recognises the country context and perceived needs, 2) an M&E 
demand study to better understand the needs, and 3) technical support to strengthen the country's capacity 
to offer training and carry out evaluations. This strategy has not yet been fully implemented. 
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Viability" 


Finding 75: Despite the challenges encountered by the FA centre to date, there is reason 
for some optimism as regards its viability. This is, however, likely to require the 
continued availability of funding from CLEAR or other donors for M&E related 
capacity building in the region in the mid to longer term. 


This finding is based on the following observations: 


Planning for financial viability: During the evaluation site visit, CLEAR FA staff discussed the notion 
of centre sustainability primarily in terms of financial sustainability, in particular how to ensure that 
training courses could pay for themselves. The sustainability or business plan will be done only 
when the strategic plan is finalised. 148 


Integration into host institution: CESAG senior leadership has been supportive of CLEAR and 
remains committed to making it work. The DG made a strong statement during the debriefing of the 
consultants in February 2014 to the effect that the six CESAG-funded positions were in place in the 
CLEAR centre and would remain so. Overall, while the two entities have not yet worked as closely 
together as they could, the recent changes in CLEAR FA personnel and the creation of a CESAG 
director of research may change this. 


The FA centre is aligned with the objectives of CESAG and could make a significant contribution to 
the goal of CESAG becoming a world-class management school. CESAG expects that CLEAR will 
play a major role in launching CESAG’s envisaged M&E master’s programme in 2015. This 
confirms the longer-term interest in and commitment of the institution to strengthening M&E 
capacity, and to doing so in collaboration with CLEAR FA. 


Context: On the positive side there is strong demand for M&E capacity development services from 
the government of Senegal, which puts particular emphasis on conducting more training of 
trainers, offering training sessions for evaluators, and helping change the broader culture for M&E 
within government systems. The governments of Benin and Burkina Faso have also expressed 
interest in obtaining CLEAR services to support their public service reform programmes. While it is 
uncertain what types of funding these governments will be able to mobilise, it is likely that they will 
be able to generate at least some funds (e.g. through donor support to public sector reform) to pay 
for CLEAR services. It is, however, unlikely that clients in the region will in the short or even mid- 
term be able to make such funding 


available from internal resources. CLEAR FA centre affiliation with the World Bank 


The existing demand for the types Several respondents noted that in their experience the influence of 

of services provided by CLEAR FA | the World Bank was an important, if not the main driver for their 

appears to be closely related to its respective governments to not only seek support for M&E/RBM 

affiliation with the World Bank (see capacity development, but to do so by contacting the CLEAR FA 

sidebar). centre. The continued reputational affiliation of the centre with the 
WB was therefore seen as an important factor for sustaining existing 

There are, however, also a number | demand for its services from national governments. 

of (actual or likely) contextual 

challenges that can affect the centre’s continued viability: 





e CLEAR FA is situated in a teaching institution which is recognised for its training 
capabilities but CLEAR’s mandate is broader than simply training. Training is, however, 
only the first step in providing a technical package to other organisations. At present, 


145 In the context of the CLEAR centres, we understand the notion of (centre) viability to encompass i) 
financial viability; ii) institutional viability, and iii) contextual viability, i.e. continued relevance of centre 
services. 


146 Expected for the fall of 2014. 
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the CLEAR FA centre does not yet have sufficient internal or external capacity 
(including through CESAG staff) to provide, for example, other forms of support such as 
mentoring and strategic planning. 


Another potential challenge in view of planning for financial sustainability is the fact that 
in the cultural heritance of Senegal and the whole region education is considered a 
right, i.e. it has traditionally been free. This conflicts with the idea of education as a 
privilege, which would be implied by offering courses (be it the envisaged M&E masters 
program, or one-off training courses) for a fee. 


3.6.5 Forward looking considerations 


The following suggestions for consideration by the CLEAR FA centre derive from the findings 
outlined above. 


As the FA centre finalizes its draft strategy, we encourage it to ensure that this 
document elaborates on the centre’s understanding and vision for what ‘success’ will 
look like in its particular case. More specifically, what types of short and midterm results 
is the centre hoping to (and realistically able to) contribute to at national (Senegal) and 
regional levels? What capacity it is trying to build, for whom and how? What will be 
suitable indicators (or markers) of progress towards these objectives? 


Related considerations may start with a review of language. Capacity is often used 
across a wide range of levels of activities from the individual to the sector although it 
takes different forms in each case.'* The information elicited in the country demand 
studies conducted by the centre provides some first, broad ideas in this regard. 
However, specific decisions on capacity development interventions will likely need to be 
discussed with individual clients in order to determine, for example: What goals is the 
desired capacity intended to support? Where should this capacity be located: with 
individuals, groups, or broader organisations? How would activities for these different 
audiences interrelate and reinforce each other 


Another consideration that is related to implementing the noted 3-step process for 
working with countries other than Senegal is in how many countries the centre wants to 
(and can) work, and why. This may include the development of criteria to determine 
which countries it will (or will not) engage in.'*® The centre should make explicit how, if 
at all, its approach in Senegal will differ from the one taken in other countries. 
Furthermore it would be helpful to address the envisaged role of partner organisations 
in realising the centre’s objectives in Senegal and/or the wider region. 


We commend CESAG and the CLEAR FA centre on its recent and ongoing efforts to 
reduce the burden of administrative and financial regulations that have, in the past, 
posed difficulties to the work of the centre. If and as needed, we encourage the CLEAR 
FA centre and CESAG to approach the CLEAR Secretariat for advice or technical 
assistance in this regard if and as needed. Such support may, for example, be helpful 
for addressing noted concerns over existing regulations being solely focused on 
ensuring accountability, rather than on facilitating and capturing the centre’s 
achievements. 





147 For example at the individual level, competencies or skills and motivations are important. At the 
organisational level, the ability to carry out day-to-day duties such as logistics, management and technical 
tasks are important. For more details on the notion of “capacity” and “capacity development” please see: 
Capacity, Change and Performance (2008). Maastricht: European Centre for Development Policy 
Management. 
148 E.g. should it engage in fragile states, or not? The criteria developed by the Anglophone Africa centre 
may provide inspiration in this regard. 
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e There is a need to identify what types and numbers of professional capacity are needed 
to implement the envisaged centre strategy — both in terms of internal centre capacity, 
as well as access to specific external resources, be it CESAG faculty, or other 
organizations. Consolidating existing partnerships and systematically exploring new 
(purposeful) ones will be beneficial in this regard. 
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4 CLEAR - Global Learning Component 


4.1 Introduction 


As per CLEAR strategy (2013-2018), the global learning component aims to “generate, aggregate, 
and disseminate global knowledge and peer-learning about what works, what doesn’t, and why, to 
strengthen the expertise of the CLEAR Centres and communities of practice on M&E across 
regions”. Specifically, the component aims to: 


e Develop and share global knowledge and expertise, in particular by identifying and 
filling gaps in knowledge at the regional level, 


e Facilitate peer-learning through the global network, by facilitating knowledge exchange 
and providing mentoring across regions. 


Approximately 10% of the overall CLEAR budget is devoted to global learning.*** The CLEAR 
Secretariat is responsible for planning and implementing the global component. To date, global 
component planning takes place annually as part of CLEAR’s annual work plan process and 
identifies activities and/or outputs planned for the upcoming period (including publications, training 
modules, the Global Forum among others). When applicable, Regional Centre work plans and 
budgets also identify and set aside some resources for global component activities/outputs (e.g. 
learning products). 


This section explores the performance of the global learning component of CLEAR in terms of its 
design and effectiveness to date. 


4.2 Design 


Finding 76: The global learning component was congruent with CLEAR’s vision, objectives, 
and experimental nature, but the lack of an articulated strategy has limited the 
potential learning opportunities deriving from this component. 


CLEAR’s global learning component made sense for several reasons. First, in light of the 
experimental nature of CLEAR, it provided dedicated financial resources that could be used to test 
implicit and explicit ‘developmental’ assumptions embedded in the CLEAR Theory of Change, 
including for example; 


e |f, how, and under what conditions Regional Centres could support the realisation of 
CLEAR outcomes, namely: to help improve the enabling environments and demand for 
M&E; strengthen capacity to produce and use evidence; expand professional M&E 
expertise in regions; and support innovation in M&E 


e |f, how, and under what conditions CLEAR outcomes would lead to higher level 
outcomes (strengthened monitoring and evaluation systems and practices) 


e If, how, and when CLEAR’s higher level outcomes would lead to the highest level 
outcome (stakeholders use evidence in making decisions for improved development 
results). 


149 Source: CLEAR Strategy 2013-2018, p. 4. According to the CLEAR Annual Report July 2012-June 2013, 
actual expenditures at that point for the Global Component constituted 14% of overall resources. This 
included, however, expenditures for Network Support (Website/Communications, and Task Forces on 
Governance and Quality Assurance). 
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Second, it provided stakeholders with opportunities to invest in the CLEAR objective to support 
regionally grounded M&E capacity that would be informed by "global knowledge" and to 
systematically examine other aspects of the CLEAR design, including for example: 


e The conditions under which a group of geographically disperse entities with dramatically 
different contexts and managerial and professional capacities (the Regional Centres) 
could form a functioning, value-added and sustainable network 


e The appropriateness of the way the programme was governed at global, regional and 


national levels. 


However, CLEAR has not yet 
formulated an explicit strategy that 
articulates what the global 
component is hoping to achieve, 
which makes it difficult to 
understand if and how the planned 


Outcomes of Global Learning as per CLEAR results framework 
CLEAR global knowledge 


Peer learning 


activities/outputs complement one another and how they are expected to contribute to higher level 
results. The outcomes for the global learning component outlined in the CLEAR results framework 
(see sidebar) fail to describe the envisaged expected changes normally found in results 
statements. The related indicators focus on the extent to which learning is taking place within the 
network by 2018 (from Centre to Centre and/or through the network of centres) and the extent to 
which CLEAR is recognised as a global brand by 2018. These indicators appear more appropriate 
for measuring the performance of an entity that has established capacity in generating knowledge, 
rather than for an experiment that is as yet unproven and whose potential relevance and added- 


value is being examined. 


Moreover, the term “global 
knowledge” has not yet been 
defined in the context of CLEAR, 
which has added ambiguities and 
resulted in a fragmented global 
learning component. The notion of 
aggregating and sharing ‘global 
knowledge’ implies a value-added 
function brought about by reflection 
and analysis in order to translate 
context-specific experiences into 
higher level insights. This is not yet 
strongly visible in CLEAR annual 
reports which have provided little or 
no analysis of experiences or 
learning gained across centres. 


Similarly, the publications available 
on the CLEAR global website do not 
yet reflect a shared vision of the 
types of knowledge and learning 
that CLEAR aims to contribute to. 


Publications available on the CLEAR website ‘°° 


Demand and Supply: Monitoring, Evaluation, and Performance 
Management Information and Services In Anglophone Sub-Saharan 
Africa A Synthesis of Nine Studies (2013) 


Exploratory Case Studies. African Monitoring and Evaluation 
Systems (2012) 


Workshop Report: African Monitoring and Evaluation Systems 
(2012) 


An Assessment of Demand for Monitoring, Evaluation, and 
Performance Management Information and Services (Benin, 
Senegal, and Mauritania) (2013) 


Mobile-Based Technology for Monitoring and Evaluation (2013) 


From Recommendations to Actions - Federal Programs Committed 
to the Evaluation Process (2011) (available in Spanish only) 


Embracing Evaluative Thinking for Better Outcomes: Four NGO 
Case Studies (undated, added to the website in mid-2014) 


CLEAR Global Forum Report — Mexico City. November 2013 (added 
to the website in mid-2014) 


Three publications are related to the demand study conducted in Anglophone Africa; two 
publications address, more or less directly, the question of ‘what works, what doesn't, and why’, 151 


150 http:/Awww.theclearinitiative.org/clear_pubs.htm! 
151 ‘From Recommendations to Actions’ and “Embracing Evaluative Thinking’. 
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but one is available only in Spanish. In addition, several of these publications tend to be more 


descriptive than analytical. 


4.3 Effectiveness 


This section examines the effectiveness of the global learning component in terms of its objectives 
to strengthen the M&E capacity of the regional centres and facilitate peer learning through the 


CLEAR network. 


Finding 77: CLEAR regional centres have benefited, to varying degrees, from Secretariat 
support and CLEAR training modules provided as part of the global learning 
component. It is not clear to what extent or how this has strengthened their 


internal capacity. 


The document review and consultations with Secretariat and regional centre staff indicate that all 
centres — to varying degrees — have made use of and benefited from one or more services or 
products made available as part of the global component. These are discussed below. 


Coaching/mentoring from the CLEAR Secretariat: All centres have benefited from support 
provided by the Secretariat in the form of advice, networking, and assistance in administrative and 
professional issues. In some cases (e.g. the AA centre) the Secretariat has also provided direct 
assistance to centre clients, thereby complementing the centre’s own professional capacity. The 
extent and types of support requested varied considerably among the centres, reflecting their 
professional and managerial experience at the outset of CLEAR. The FA and AA centres 
requested support from the Secretariat most frequently, while the LA and SA centres have done so 
only occasionally and increasingly less over time. Secretariat support to the centres has been 
responsive and demand-driven (i.e. based on requests). The evaluation found no evidence of 
discussions or a shared understanding within CLEAR on how the Secretariat should navigate the 
delicate balance between providing appropriate and constructive support to strengthen centre 
capacity on the one side and fostering dependency on external help on the other side. 


Training modules: The Secretariat developed a series of training modules for use by the Regional 
Centres.**? The topics were based on consultations with regional centre staff who were asked to 


identify key issues that were 
relevant in their contexts and in 
which they lacked knowledge and 
skills. Centre staff and clients in AA, 
EA, and SA acknowledged that the 
training modules were well- 
researched and reflected current 
international knowledge on the 
topic. The centres in Anglophone 
Africa and East Asia have been the 
most frequent users of the training 
modules to date.*** Factors that 
affected the centres’ use of training 
modules are described in the 
sidebar. 


Factors that affected the use of CLEAR training modules 


Whether the regional centre already had training modules 
addressing the same or similar content — This was the case, for 
example, in South Asia where J-PAL already had a well-developed 
module on Impact Evaluation 


The perceived relevance of and demand for the module in the 
different contexts - Persons delivering the Impact Evaluation 
module in AA questioned its appropriateness in several African 
countries given modest M&E capacities and limited government 
demand and resources for such evaluations at the time. 


The stage of centre development — The Francophone Africa 
Centre has not yet used any of the modules. Consulted centre staff 
largely attributed this to the fact that the centre had only been 
operational for a limited amount of time. 


152 Topics covered to date are: Introduction to M&E; Impact Evaluation; and Performance-Based Budgeting; 
a fourth module on Rapid Evaluations is under development 


153 In 2012-2013, the module on Impact Evaluation was delivered in Anglophone Africa (South Africa, and 
Uganda) and East Asia (China) only, but not in the other regions. 
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Staff in the AA and EA centres noted that the training modules allowed them to offer high-quality 
training courses very soon after being established, thus saving time and effort that would have 
been required to develop similar modules. Centre staff and clients of the EA and AA centres 
commented that involving international experts in the development and delivery of the modules 
raised the perceived profile of CLEAR’s services. 


The training modules provided by CLEAR allowed regional centres to provide some well-regarded 
and high quality training in the short term. The extent to which these will be used in the future is not 
clear and varies by centre. The SA centre no longer sees a use for the modules, and the EA centre 
remains strongly dependent on international experts to conduct the training. The AA centre has 
adapted the modules for its context, frequently engaging regional associates to work alongside 
international associates to contextualise the training for participants. 


Finding 78: The global learning component has facilitated some opportunities for South- 
South exchange and peer learning, especially through the annual Global 
Forum. CLEAR units are still in the process of developing a shared 
understanding of the specific intended purposes and the best processes for 


peer learning. 


Centre staff from all regions acknowledged that CLEAR, and the Secretariat in particular, has 
encouraged and facilitated exchange among the centres and with some of their partners. Key 
elements in this regard have been the annual CLEAR Global Forums, quarterly conference calls 
among centre heads (recently initiated and led by the Secretariat), and increasingly initiatives 


driven and led by the centres 
themselves. 


The CLEAR Global Forum: Most 
consulted stakeholders who had 
attended the Global Forums (i.e. 
Board members, Secretariat staff, 
centre staff, and their clients and 


collaborators) had positive views on 


the relevance and utility of the 
events which they considered 
valuable opportunities to meet with 


Participant appreciation of the Global Forum 


“The Global Forum is a great opportunity for networking and 
learning. It is extremely interesting and relevant to hear what the 
other centres are doing and how.” (RC staff member) 


“The Global Forum provides unique opportunities for networking. 
For example, | was able to meet and speak with a high level 
government representative from my own country, whom | would not 
otherwise had access to.” (NGO Representative) 


and hear about the work and experiences of other regions, as well as for networking (see sidebar). 


All consulted stakeholders agreed that having a CLEAR regional centre take the lead in organising 
the global forum in Mexico in November 2013 was a promising development that fosters centre 
leadership and ownership not only for event logistics but also for the contents addressed at the 
forum. They also commented positively on the increased involvement of CLEAR clients and 


collaborators in the Mexico event, 
which was widely seen as an 
improvement over earlier forums. 


At the same time, several 
respondents (in particular those 
who had attended two or more 
forums) noted that the forum had 
not yet lived up to its potential to 
become a platform to collectively 
“push the global envelope”*** in 
relation to M&E capacity 


154 Consulted CLEAR regional partner. 


Global Forum -— Participant comments 


“| attended one of the first global forums where we had excellent 
discussions on the topic of Impact Evaluations and Randomised 
Control Trials. The debate was heated and passionate. Since then | 
feel that the Forum has become less and less a place for such 
debate — and that is a great loss, as it is this culture of debate that 
can help us to collectively push the envelope. Now it seems to me 
that the forum is all about consensus and administrative issues — 
that is a lost opportunity. In my view CLEAR should make the event 
into the equivalent of the World Economic Forum but for M&E.” 
(CLEAR Partner) 
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development or promoting the use of evidence. An illustrative comment is shown in the sidebar 
above. This links back to the noted absence of clearly defined learning objectives that CLEAR, 


overall, intends to contribute to. 


Other exchange and 
collaboration among centres: 
During the early stages, CLEAR 
regional centres were primarily 
concerned with getting established 
and known among stakeholders in 
their regions. While this limited the 
time, energy and resources 
available for engaging with other 
centres (outside of the Global 
Forum) there have been several 
bilateral exchanges that were 
initiated and implemented by the 
centres themselves (see sidebar). 


At the same time, CLEAR 
implementation experience to date 
illustrates that 1) networks do not 
simply emerge, but require active 
efforts, and 2) that it takes time and 
experimentation to identify issues in 
which two or more centres can learn 
from each other in ways that have 
the potential to make a difference. 
The most prominent example to 
date is that three regional centres 
(AA, FA, and SA) have expressed 


Exchange between centres to date 


In 2014 the RCs in Latin America and South Asia facilitated a visit 
of the head of CONEVAL (Mexico) to India to meet with the newly 
established Indian Independent Evaluation Office (IEO). One key 
topic of interest for the IEO was the structure of CONEVAL's 
advisory board which includes academics, i.e. stakeholders from 
outside government. Another was the influence of the enabling legal 
environment in Mexico on progress in that country. First contact 
between the CONEVAL and IEO leads had been established at the 
CLEAR Global Forum in Mexico. 


The Latin America and Anglophone Africa centres worked 
together to assist the South Africa DPME in accessing expertise 
from Colombia — with the purpose of learning from the Colombian 
experience in establishing a monitoring and evaluation function with 
the federal government. Interviewed DPME representatives 
underlined the value of these contacts and subsequent interactions 
in the development of an M&E policy framework for the government 
of South Africa. The same centres collaborated on co-organising 
South-South roundtable discussions in Pretoria on government M&E 
systems. 


Several CLEAR centres are planning exchanges or other joint 
activities: The Francophone Africa and Latin America centres are 
exploring opportunities for CONEVAL to provide support to the 
Bureau d Organisation et Méthodes (BOM) in Senegal. The Latin 
America and South Asia centres are preparing a joint presentation 
for the next American Evaluation Association’s annual conference. 


interest in learning from the LA centre experiences in institutionalising the evaluation function, an 
area in which the LA region is seen to be advanced. Other common areas of interest may emerge 
over time, including unsolved questions or problems that all centres face and that all centres can 
contribute to addressing from different angles. 


CLEAR works in regions that are at different stages of economic and social development and that 
also vary in their approach to and capacity for results management and M&E. While this diversity 
provides an opportunity for CLEAR to learn from a range of experiences, it can also create 
challenges for facilitating meaningful and equitable exchange among the different centres. For 
example: 


e Not all centre staff and their stakeholders are comfortable conversing in English, which 
is often the common 
language at Global 


Forums “The (global) Forum in Mexico was interesting, but many of the 


contents addressed were not directly relevant to us, as most Latin 
American countries are far more advanced in M&E than countries in 
our region.” 


e Significant differences in 
the (actual or perceived) 
stage of development of 
two regions as regards 
M&E systems and 
expertise can lead to the impression that a region that is more advanced has nothing to 
learn from a region that has more basic needs. See sidebar. 


Global Forum participant from Francophone Africa 
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e The centres do not have resources dedicated to facilitating exchange between centres or 
with their affiliate centres or for translating relevant documents and tools to make them 
accessible to potential users in other regions. 


Finding 79: The CLEAR network is only emerging and it is premature to assess if and how 
the Regional Centres will decide to share knowledge and expertise in the 
future. 


In the CLEAR results framework, one of the three indicators for the global learning component is 
that “by 2018 regional centres choose to continue sharing knowledge and expertise through a 
global network”, thereby highlighting the sustainability of the CLEAR network as an explicit 
objective of the initiative. 


To date, regular exchange among the centres is still considerably dependent on facilitation from 
the Secretariat. Using a Secretariat function to facilitate exchange among network nodes is not 
unusual during early stages of network development, and many networks continue to use some 
sort of coordinating function throughout their existence.'* However, the continued existence of 
such a function in CLEAR will be dependent on resources from either external donors or the 
network members themselves. 


Some regional centres have made independent efforts (i.e. without facilitation from the Secretariat) 
to collaborate with each other. Their willingness to continue doing so, even in the (potential) 
absence of a coordinating body, is promising. However, it is not clear what, if any, resources 
different centres will be able and willing to invest in the growth and development of the CLEAR 
network once they have become fully or at least mostly independent from CLEAR funding. One, 
and maybe the most important, factor in this regard is that it is not yet evident what - other than 
CLEAR funding - forms the ‘glue’ that can continue to hold the global network of CLEAR centres 
together over time. 


Currently, the centres share similar mandates but they are only starting to explore and experiment 
with ways in which their collaboration can inform and enhance their respective capacity building 
work, or how it can otherwise benefit their clients and partners. Wind (2005) defines a sustainable 
network as one that “continues to function until it achieves its goals.” 158 At midterm, the goal(s) of 
the CLEAR network are not yet clearly defined. 


4.4 Summary/Conclusions 


While elements of the global learning component have been valued by beneficiaries (e.g. the 
Global Forum, CLEAR training modules, and the Secretariat’s support to regional centres), it is 
difficult to assess whether or to what extent the global learning component has been effective 
because CLEAR has not yet developed a detailed strategy that articulates the desired results. At 
midterm, CLEAR units are still experimenting with ways and areas of collaboration, and the 
regional centres have shown varying degrees of interest and capacity to engage in mutual 
knowledge exchange and related efforts. CLEAR has not harvested the knowledge that is being 
generated due to the absence of a knowledge management strategy that identifies learning or 
research questions. 


Overall, the global learning component has not yet realized its potential to add value to 
stakeholders, e.g. by systematically harvesting lessons and evidence emerging from the CLEAR 
experiment. This is a missed opportunity. 


155 For example the Consultative Group on International Agricultural Research (CGIAR). 


156 Wind, Tricia. 2005. Evaluation Highlight 3: The Sustainability of IDRC supported Networks. IDRC 
Evaluation 5pp. 
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5 Future Directions 


5.1 Concluding Remarks 


The CLEAR vision and related objectives (to anchor development in evidence, learning and mutual 
accountability) are highly relevant to the global discourse on results-based management and aid 
effectiveness, and increasingly relevant to the values and policies of a growing number of countries 
in all regions in which CLEAR is present. Despite critical findings in this report, the evaluation team 
sees considerable value and potential in pursuing the CLEAR vision, given the increasing demand 
for and limited supply of solutions to enhance the use of evidence in decision making. While 
CLEAR achievements to date are generally modest in terms of what was envisaged in CLEAR’s 
programme design, this is to be expected in what is still a relatively young, experimental initiative 
that had to build a programme architecture before it could commence most of its planned activities. 


Throughout the course of this evaluation, we have seen positive, encouraging signs of CLEAR 
stakeholders’ belief, enthusiasm, commitment and support for the ideas and aspirations behind 
CLEAR. At this point, the evaluation team sees the value in continuing the CLEAR experiment, but 
with an approach that is considerably more focused on capturing, sharing and utilising the learning 
that is taking place, so that lessons and insights can be used to inform CLEAR’s future directions, 
and ideally inform other similar capacity building initiatives. 


In the remaining years of the first phase of CLEAR, it will be important that CLEAR activities be 
better grounded and connected to regional and host institution realities and contexts, and 
appropriately designed to respond to such contexts. Furthermore, better/more effective 
engagement with CLEAR’s regional stakeholders can also simultaneously pave the way for greater 
regional ownership and sustainability of CLEAR results. Finally, it will be important that the roles 
and responsibilities of the CLEAR Board and Secretariat be better aligned with these new 
directions. This will require some changes in how CLEAR is conceptualised, managed, governed, 
staffed, implemented and monitored so as to maximise the potential of the initiative. 


5.2 Recommendations 


This section provides some constructive, practical recommendations for the time remaining in the 
current phase of CLEAR (i.e. to 2018). The recommendations and suggestions for their 
operationalization are presented in sections related to the overall CLEAR programme, the regional 
component (i.e. the regional centres), and the global learning component. 


5.2.1 Overall Programme 


Recommendation 1: The CLEAR Board should decide if CLEAR is an experiment that is 
primarily intended to generate lessons learned that can inform future phases of CLEAR and 
other (M&E) capacity building initiatives. The Board should approve a set of overarching 
learning questions that CLEAR seeks to answer as well as the types of information that it 
will monitor at the level of the initiative. 


A couple of the key findings emerging from this evaluation relate to the disconnect between the 
experimental nature of CLEAR and how it has been managed to date, and some mixed messages 
about its intent (i.e. whether it is an experiment focused on learning what works, when and why OR 
a programme that is accountable for specific development outcomes.) In our opinion, CLEAR is an 
unfinished experiment and it is critical for the Board to clarify CLEAR’s status so that all 
stakeholders (particularly Board members) share the same expectations. In particular, the CLEAR 
Board needs to: 
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1) Approve the overarching learning questions that the CLEAR initiative as a whole is aiming to 
explore. These questions should address two areas: 


e Learning about what is working and not working with the CLEAR design and 
implementation; 


e Learning about the change processes and factors influencing the use of M&E- 
generated evidence in decision making. 


The learning questions should be 
derived from CLEAR's current 
Theory of Change, which we 
suggest remain as is for the time What are the benefits and limitations of: 

being. The questions should - centre exchange/collaboration and forming a sustainable network? 
address the assumptions about 
causal linkages for which there is 
not yet any strong evidence (e.g., - regional expertise (e.g. pressures to bring in international experts) ? 
issues noted in the sidebar, as well Questions focusing on development processes 

as under finding 5 (section 2.2). 


Questions focusing on the functioning of CLEAR: 


- centres hosted by academic institutions? 


What are the roles of different types of change agents (e.g. NGOs) 


As further elaborated in in bringing about the desired changes at organisational, sector, or 
recommendation 8 below, the (sub) national levels? 
CLEAR Secretariat, which has How can M&E influence decision making? What role does capacity 


responsibility for the global learning (development) play in this regard? What goal(s) is the desired 


component, should play a key role in | Capacity intended to support? Where should this capacity be 
defining the learning questions located: with individuals, groups, or broader organisations? How do 
systematically monitoring CLEAR activities for different audiences interrelate and reinforce each 


; ; other? 
assumptions and capturing, ; E . L 2 es 
analysing and sharing information What is the meaning and role of “innovation” in M&E in different 


suitable for (contributing to) contexts? Is innovation necessarily positive? 
answering the learning questions. 


2) Approve the types of information most relevant for capturing progress of CLEAR overall in 
light of the established learning questions. 


e The current CLEAR results framework is not well suited for capturing development 
progress and related learning occurring in the diverse contexts of the regional centres. 
This will be even more pronounced if, as suggested in recommendation 6, each of the 
regional centres develops its own vision of what ‘success’ will look like in its region. In 
our view, monitoring the success of CLEAR overall will be most meaningful if it captures 
two types of information on the regional centres: a) evidence of learning that has 
occurred in each of the centres in relation to the overarching CLEAR questions; and b) 
evidence of progress along the individual pathways of change defined by each of the 
centres. The latter is significantly different from asking all centres to report against the 
same development results. 


e We acknowledge that existing accountability requirements of CLEAR’s financial 
partners may require the initiative to have at least a few common indicators in place 
(e.g., on financial information) that all centres have to report upon. Nevertheless, we 
assume that there is some room to modify the way in which the centres capture and 
report on their programming, on development results, and related learning. 


e Finally, in clarifying the types of information to be collected in future, the Board in 
association with other stakeholders should revisit the continued appropriateness of 
indicators in the overall results framework related to the establishment and functioning 
of the regional centres and the CLEAR network. Given CLEAR's experimental status, 
we question the implicit/explicit emphasis to date on sustainability of the centres and/or 
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the network; in our view, stakeholders’ attention should instead focus more (or instead) 
on learning from CLEAR experiences to date in trying to institutionalise the centre and 
the network. 


e As for the previous point about indicators, the Secretariat should be tasked with 
developing suggestions for a revised, meaningful approach to monitoring CLEAR 
overall in light of its renewed learning focus. 


Recommendation 2: For the duration of the CLEAR experiment to 2018, the CLEAR Board 
should limit management and governance changes to those that will help transform the 
CLEAR programme into a strategically poised, learning initiative. The CLEAR Board should 
approve the establishment of an advisory committee and the appointment of a senior 
advisor to oversee the proposed transformation of the CLEAR programme into a learning 
initiative. 

The 2012 governance study commissioned by the CLEAR Board (conducted by AccountAbility) 
recommended several changes in CLEAR's governance structure to enhance its legitimacy, 
strategy, performance, and the efficiency of CLEAR. Suggestions included, for example, the 
creation of Investment and Executive Boards, among others. Other management changes 
suggested by consulted Board members during the course of this evaluation included the creation 
of a CEO position to manage the initiative. 


These suggestions would be relevant if CLEAR were already an established and proven initiative, 
seeking ways to become a more permanent entity. However, since CLEAR is still in the middle of 
an experimental phase, it is important that the CLEAR Board be pragmatic and realistic given that 
there are only three years remaining and limited resources. In our view, the Board should limit the 
number of changes to its governance and management structures as proposed below. 


1. Approve the establishment of a knowledge management committee to be overseen by the 
Board, but involving external thought leaders: To enhance the Board’s and the Secretariat s 
capacity to support learning about questions of strategic interest, the Board should approve the 
establishment of a Knowledge Management Committee as an advisory body that would assist 
CLEAR in articulating the learning questions for the period to 2018. It would also provide advice 
and support for developing a knowledge management strategy. The committee, which would be 
chaired by a CLEAR Board member 

and staffed with a senior knowledge 


management advisor (see below), Knowledge management committee — member criteria 
should comprise a small number (3- Solid subject matter expertise on development evaluation and/or 
5) of carefully selected global and development evaluation capacity building 


regional thought leaders. Building on 
suggestions in the AccountAbility 
study, the members of the Excellent reputation and respect among CLEAR’s key stakeholders 
committee should have some of the Deep regional knowledge of the development evaluation landscape 
characteristics shown in the sidebar. | of one or more of CLEAR's priority regions 

Committee members should be Strong knowledge of and experience in how to enhance demand for 
selected for their expertise, track M&E 

record, interest and commitment to 

participate for a three year period, rather than because of the groups they represent. The 
committee should meet at least 4 times per year in the first year, and 2-3 times per year 
subsequently. In keeping with CLEAR Board members’ current practices, knowledge management 
committee meetings could be held virtually, and/or in tandem with planned global/regional M&E 
events in order to contain the costs of such meetings. 


Organisational leadership and management experience 


2. Approve the appointment of a knowledge management advisor: The Board should recruit or 
second an experienced, senior thought leader as a knowledge management advisor who could be 
housed in or contracted by the CLEAR Secretariat to oversee the design and management of the 
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proposed reformed global learning component described under Recommendation 8. The 
knowledge management advisor would report regularly to the CLEAR Board through the 
knowledge management committee. He or she would work in tandem with CLEAR Secretariat and 
regional centre staff to carry out the proposed research. 


These suggestions have some financial implications and will likely require the re-allocation of some 
financial and human resources to allow strengthening the CLEAR global learning component, and 
perhaps some additional resources (from current financial partners or other sources). 


Recommendation 3: THE CLEAR Board should assume a more strategic role in the future, 
deferring more operational considerations to the CLEAR Secretariat. 


The evaluation noted stakeholder concerns about the limited attention paid by the CLEAR Board to 
strategic issues; similar issues were identified in the 2012 AccountAbility study, which also made a 
number of useful recommendations. 


In keeping with several suggestions in the AccountAbility study, we suggest that the CLEAR Board 
assume greater responsibility for managing the initiative more strategically in the future. In 
consultation and with support of others as required (including the RCs, the RACs, the CLEAR 
Secretariat, and the proposed knowledge management committee), it should: 


e Take a leadership role in transforming the CLEAR focus on learning — this includes the 
appointment of a Knowledge Management (KM) committee, appointment of KM committee 
members, appointment of a KM advisor, and adoption of a KM strategy that would clarify 
what processes will be used to capture, develop, share, and effectively use knowledge 
within (and beyond) CLEAR, as well as related roles and responsibilities. 


e Approve CLEAR's vision and strategy to reflect the increased attention on learning 


e Provide guidance to the Secretariat on key parameters for revising the current CLEAR 
results framework in order to develop an approach to performance monitoring and reporting 
that are aligned with the experimental nature of CLEAR as suggested in 
Recommendation 1. 


e Advise the Secretariat on key management and administrative decisions and processes 
such as strategic planning, budgeting and budget review, external positioning, awareness 
building, and fundraising. 


e Promote knowledge sharing and collaboration among relevant evaluation practitioners 
within and external to the CLEAR initiative. 


Recommendation 4: The CLEAR Board should not consider any further expansion into new 
regions between now and 2018, and instead focus its efforts and resources on 
consolidating the learning taking place in the current CLEAR centres. 


Although a couple of CLEAR centres (LA and SA) are now well established in their host institutions 
and actively engaged in CLEAR programming in their regions, they still need more time to test the 
assumptions imbedded in the CLEAR design and (ideally) begin to realise envisaged outcomes. 
Other CLEAR centres are at much earlier stages of institutional development (AA and FA) and/or 
in undertaking the kinds of programming envisaged in the CLEAR design (EA and FA). 


In our opinion, the CLEAR Board should not create any new CLEAR regional centres between now 
and 2018 but should concentrate resources on the ongoing CLEAR experiment and learn the most 
from each of the six already approved regional initiatives (including the one in Brazil). 


5.2.2 Regional Component 


For each of the five reviewed regional centres, a number of forward looking suggestions were 
noted in chapter 3 of this report. The following recommendations address common issues and 
areas for improvement. 
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Recommendation 5: All CLEAR centres should establish and operationalise Regional 
Advisory Committees (RAC) by December 2014. 


The absence of functioning regional advisory committees deprives four of the reviewed CLEAR 
centres from some potentially valuable strategic guidance that could guide them in developing 
relevant strategies and programmes. It also reduces opportunities to strengthen regional 
ownership of the centres and their work. 


Thus we strongly encourage each centre to expedite the establishment and/or functioning of its 
RAC by December 2014. In cases where RAC membership is not yet finalised, we suggest that 
each centre should decide as soon as possible whether its committee should be regional or 
national in nature. While regional representation is desirable, national representation should be an 
option if that makes sense for the centre in view of its (evolving) region-specific vision. The CLEAR 
Board may provide advice on request but should not have any say in RAC membership. 


Recommendation 6: Each CLEAR centre should develop a centre-specific theory of change 
(or at least elements of such a theory) to clarify key ideas and assumptions on the purpose, 
priorities, and envisaged results of the centre. Theories of change (ToCs) may differ in their 
format and level of elaboration, depending on existing centre capacity. 


Each of the centres should be encouraged and supported by the RACs to elaborate their thinking 
and assumptions regarding the change processes they are aiming to influence in their region (or 
country), how their activities/outputs are envisaged to contribute to broader changes, and what 
specific niche and role(s) the centre aims to fulfil. Considerations on how to operationalize this 
recommendation are outlined below. 


e In recognition of variations in CLEAR Centres’ contexts and institutional development, 
CLEAR Centres should be encouraged and supported by the Secretariat as required to 
create or adapt existing Centre-specific theories of change over the next nine months 
(to June 2015). The process can be a gradual one, where Centres continue with their 
current strategies, while also identifying (new) CLEAR learning questions, and use 
related insights to elaborate on the desired results in their regions (i.e., what it is they 
are trying to change). 

e |n some cases, developing centre-specific theories of change may merely require 
articulating the centre’s already implicit assumptions about how its activities will 
contribute to broader development objectives, and/or finalising a region-specific ToC (or 
results framework) that is already underway. 


e ToCs should be realistic and matched to the centre’s existing (and likely future) 
capacity. Depending on staff members’ experience, some centre teams may require 
Secretariat assistance in developing ToCs. While the ToCs of different centres may be 
at different levels of detail and sophistication, the aim should be to use plain, clear 
language, and articulate all assumptions. Regional Advisory Committees should 
contribute to/provide feedback on this process. 


e The individual centre ToCs should be linked to the overall CLEAR ToC and/or the 
overarching CLEAR learning questions (see Recommendation 1). However, instead of 
focusing on “making centre activities fit” into this overall ToC, emphasis should be 
placed on realistically describing the types of changes that the centre is likely to 
contribute to in the midterm, and/or the main questions that it is hoping to gain insights 
into. Some centres may decide that they can realistically contribute to only one or two of 
the development outcomes in the current (overall) CLEAR ToC or to only few of the 
(suggested) CLEAR learning questions, while others may aim to contribute to all of 
them. 


e Deliberations on the centre’s envisaged niche and role(s) requires clarifying the role 
that other change agents are already playing or are envisaged to play in this regard, 
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both in collaboration with CLEAR and by themselves. This will also help clarify what 
types of actors each centre considers to be ‘strategic’ clients or partners. 


e Some centres may decide to focus (primarily or exclusively) on the national or sub- 
national level, and considerably limit or even eliminate regional engagement. 


e Each centre should identify what (sub) national capacity and/or strengthened M&E 
systems will look like, and the milestones (progress markers) that should occur on the 
path towards achieving these objectives. In addition to (or possibly instead of) tracking 
traditional technical indicators (such as ‘stakeholder knowledge of M&E/RBM 
approaches and tools’) centres may also want to look for evidence of changes in client 
organisations as a result of their enhanced knowledge (such as changes in the client’s 
organisational policies, practices, resource allocations, and so forth). 


e The theories of change should be aligned with the each centre’s business and 
sustainability plans where these exist. 


Recommendation 7: Until 2018, the CLEAR Board should approve support for CLEAR 
centres’ growth and development and engagement in the CLEAR initiative as long as their 
strategies and plans are congruent with and add value to CLEAR’s learning and 
development objectives. 


The original CLEAR design included support for Centre growth and development with full 
knowledge of the variations in the contexts and capacities of regional centres. Pressures arising 
from finite resources and calendar time may lead the Board to consider limiting or even ending 
support to those centres that have made less progress when measured against the current CLEAR 
results indicators. In our view, a more appropriate response is, however, to understand the existing 
variations in centre capacities and performance as valuable learning outcomes of an ongoing 
experiment that should be gathered, analysed and synthesised to inform CLEAR research. 


Based on this understanding, for the remaining period up to 2018 the Board should put the onus on 
the regional centres to propose what they would like to accomplish in the remaining years of this 
experiment and identify the support they would need from CLEAR to get there. Given the proposed 
enhanced focus on learning within the initiative, one of the conditions that the Board should 
consider should relate to the expected value of the research questions that the regional centres 
propose to address for the duration of the initiative. 


5.2.3 Global Learning Component 


Recommendation 8: The CLEAR Secretariat should reformulate the global learning 
component and develop an explicit strategy for the Board’s approval. 


As part of its responsibilities for the global learning component, the CLEAR Secretariat with the 
guidance of the Knowledge Management Committee and under the operational leadership of the 
Knowledge Management Advisor should identify how the global learning component will be 
reformed in order to harvest the knowledge, lessons and evidence emerging from the CLEAR 
experiment. The Secretariat also needs to lead the process of collecting information on learning 
that has occurred in the centres (and at the global level as applicable), analysing it in view of the 
overarching CLEAR learning questions suggested under recommendation 1, and sharing related 
insights within CLEAR as well as with broader communities of practice. The CLEAR Board, in 
consultation with the proposed Knowledge Management Committee, should explicitly add this 
aspect to the envisaged purpose of the global learning component. 


In addition, the CLEAR Secretariat (under the leadership of Knowledge Management Advisor) 
should develop an explicit strategy for the global learning component, that would address (but not 
be limited to) the following issues. 
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Clarify and/or articulate the role(s) and expected benefits deriving from the global 
learning component in view of a) addressing the CLEAR learning questions, b) 
strengthening centre capacity, c) facilitating networking among centres. 


Review and clarify or revise roles and responsibilities of (and within) the Secretariat and 
the regional centres respectively as regards different elements of the global learning 
component, and clarify how different types of services/contributions will lead to the 
envisaged benefits. This is of particular importance in view of the aim to collect, 
analyse, and share insights on the overarching CLEAR learning questions. 


Clarify the envisaged target groups for different elements of the global learning 
component, i.e. which, if any elements, solely focus on the regional centres, and which 
also aim to benefit other stakeholder groups, why, and how. 


Clarify the types of centre capacity strengthening services that are available from the 
Secretariat (or other regional centres). 


Consider how to further strengthen the contribution of the Global Forum to foster debate 
and learning on M&E and M&E capacity development. CLEAR may want to consider 
whether and how it wants to decreasingly use the event for showcasing CLEAR specific 
achievements, and increasingly transform it into an event that fosters discussion and 
constructive debate (and related dissent) on cutting-edge M&E thinking. If feasible, 
issues addressed during the Global Forum could be framed in light of the overarching 
learning questions that CLEAR is aiming to contribute to. 
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Annex A Terms of Reference 


Contract Number: 8572 / POR503385 


Terms of Reference and Scope of Work for the 
Mid-Term Evaluation of the CLEAR Global Initiative 


1. Introduction 


This document sets out the Scope of Work and Terms of Reference for the Mid Term Evaluation of 
the Centres for Learning on Evaluation and Results (CLEAR) Global Initiative to be conducted from 
November 2013 to May 2014. 


2. CLEAR Global Initiative - Background Summary 
Context 


Effective development is guided by evidence. However, generating and using relevant and timely 
evidence on the ground has proved to be difficult. Countries’ capacity for monitoring and evaluation 
(M&E) varies — data and useful information are often missing, credible approaches to gathering 
evidence and analysis are of uneven quality, and the systematic use of evidence for making 
decisions to drive development results happens much less often than desired. 


The focus on development results achieved renewed impetus at the Busan high-level forum of 
2011, building on the commitments made in the Paris Declaration (2005) and the Accra Agenda for 
Action (2008). Similar calls to action have increasingly come from civil society organizations across 
the developing world for regionally-owned and regionally-driven evaluation. Civil society evaluation 
organizations including regional evaluation associations, national evaluation associations, civil 
society and development organizations have reinforced the Busan message and have gone even 
further by highlighting the major asymmetries in evaluation that exists between the developing and 
developed world. 


There is also a growing recognition that conventional development evaluation needs to become 
more inclusive of and responsive to a wider range of stakeholders, as well as embrace innovation 
in, for example, the use of new technology for real time monitoring and evaluation. These forces 
together set the stage and the need for the establishment of, and expectations for the CLEAR 
global Initiative. 


Purpose 


Established in 2010 as a global programme, CLEAR was designed to be an innovative approach to 
developing country government and civil society capacity in monitoring, performance management 
and evaluation. Its overall strategy is to integrate local knowledge and experience, on-the-ground 
support, and institutional development with global public goods in monitoring and evaluation’. 


1 CLEAR’s overall outcome is to contribute to stakeholders in the target regions using evidence in making 
decisions for improved development results, through strengthening context-specific M&E systems and 
practices (CLEAR Logframe, Annex A. of CLEAR Strategy, May 2013). 
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Strategy 


The strategy encompasses customized regional approaches through working with and 
strengthening competitively selected academic institutions based in partner countries2. These 
institutions implement tailor-made strategies to develop M&E capacity in government and civil 
society within each region, drawing on local knowledge and experience. The programme thus 
intends to build capacity for sustainable impact, with the aim being that centres, and ultimately 
CLEAR becomes self-sustaining from the income generated from its clients. CLEAR also focuses 
on global public goods through identifying, generating, and sharing innovative and internationally 
benchmarked knowledge and capacity building approaches. Annex 1 to this TOR illustrates the 
CLEAR theory of change, which itself will be subject to review under the evaluation in terms of its 
relevance, logic and assumptions. 


Structure 


To implement its strategy, CLEAR is structured at two levels: 


e Regional level - competitively selected regional centres (currently in Asia, Africa and Latin 
America) provide capacity building services to a range of regional stakeholder groups in 
government and civil society. Regional Advisory Councils are being established to provide 
guidance and oversight on each Centre. 


e Global level — a global Secretariat coordinates and facilitates exchange and peer-to-peer 
learning on M&E and PM across and between regional centres. The Secretariat is currently 
housed at the Independent Evaluation Group of the World Bank. The vision is for the 
programme, and the Secretariat, to be eventually located in one of the selected CLEAR 
centres. The Board of CLEAR is also global, constituted of funding agency representatives. 


Services 


The Centres aim to provide demand-driven and cost-effective services specific to each region. 
Government agencies, civil society organizations, development institutions and donors, among 
other clients, should be able to access regionally based high quality knowledge and expertise 
through several inter-related services including customised training to meet the needs of 
practitioners, advisory services and technical assistance on the design and implementation of 
monitoring systems, the application and use of different types of evaluations, and the formulation 
and implementation of public and private sector reforms to strengthen a focus on results. Some 
Centres also aim to work specifically on evaluations through teaming up with other organizations 
and professionals to conduct evaluations also serves to expand the pool of professional evaluators 
in the region. Each centre and CLEAR globally aims to build knowledge and skills, support 
communities of practice and foster leadership for monitoring and evaluation. 





2 CLEAR competitively selected the University of Witwatersrand in South Africa to host the center for 
Anglophone Africa, the Centre Africain d'Etudes Supérieures en Gestion (CESAG) in Senegal to the host 
center for Francophone Africa, the Abdul Latif Jameel Poverty Action Lab at the Institute for Financial 
Management and Research in India to host the center for South Asia, and the Centro de Investigacion y 
Docencia Económicas (CIDE) in Mexico to host the center for Spanish-speaking Latin America. The Asia 
Pacific Finance and Development Center (AFDC) in Shanghai, China, is also part of the CLEAR program. 
Expansion of the Initiative is being considered to Brazil and the Pacific. The results of this evaluation are 
expected to inform further expansion of the Initiative. Further information on the establishment of the centres 
can be found at: www.theclearinitiative.org 
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Financing 


Financial support is provided to CLEAR by: the African Development Bank (AfDB), the Australian 
Agency for International Development (AusAid), the Asian Development Bank (ADB), Belgian 
Development Cooperation, the Inter-American Development Bank (IDB), the Rockefeller 
Foundation, the Swedish International Development Agency (SIDA), the Swiss Agency for 
Development Cooperation (SDC), the UK Department for International Development (DFID), and 
the World Bank Group (WB). The Secretariat is housed in the Independent Evaluation Group (IEG) 
of the World Bank Group, and the World Bank is the program’s financial Trustee and program 
manager. 


Governance 


The CLEAR Board is constituted of the representatives of the donor financing agencies, with World 
Bank acting as the Chair and Secretary. Regional Advisory Committees are being established in 
each region to provide guidance and oversight over each regional Centre, which are managed by 
the host institutions which in turn have their own governance arrangements. 


Timeframe and Sustainability 


CLEAR was designed as a five-year programme. It is envisioned to run until 2016/17 with a 
possible extension. The selection of centres was phased, with the first selected in December 2010 
and the most recent in January 2012. The establishment of these centres has thus also been 
staggered based on these variable selection dates, while consolidation is expected to be smoothed 
once centres are up-and-running. 


It is intended that CLEAR expands its operations both within the regions in which it is currently 
focused, and, where appropriate, into new regions. The overall sustainability of CLEAR over and 
above intra-Centre continuity will depend heavily on the value of the inter-Centre relationships, and 
the global value that the initiative can bring. These issues will be looked at in the evaluation. 


4. Purpose, Uses and Users 


Requirements for a mid-term and final evaluation have been built into the funding agreements and 
expectations of funders and CLEAR Centres since the inception of CLEAR. The proposal to 
commission a mid-term evaluation of CLEAR reflects a demand by the CLEAR Board to have an 
independent assessment of the progress made at the mid-point in its lifecycle for the following 
purposes: 


1. Learning for improvements in the rationale, design, management, implementation and 
governance of the CLEAR Global Initiative. 
2. Accountability to the current funders of CLEAR for funds invested in CLEAR. 


3. Asa public good contributing knowledge on approaches to strengthening evaluation 
capacity in developing countries, designing and managing global initiatives. 
The uses and users (recipients) of the evaluation will be: 
1. The Board of CLEAR — through the evaluation findings related to governance and strategic 
oversight of the direction, performance and possible expansion of the CLEAR Initiative. 


2. The CLEAR Secretariat at the World Bank — in terms of findings on the management of the 
CLEAR Initiative and their dialogue with Centres. 
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3. Each of the CLEAR Centres’ (Asia, Latin America, Africa) regarding the performance of 
ongoing operations and development and lesson for improvement. 


4. Each CLEAR Donor (Banks, Bilateral Agencies, Foundations) — in terms of informing 
decisions on continued financing. 


5. Private sector actors, Governments and voluntary organizations of professional evaluators 
(VOPEs) - on lessons in strengthening demand and supply of evaluation capacity. 


5. Evaluation Objectives 


The specific objectives of the evaluation will guide the users and the evaluators in terms of the 
shape and content of the evaluation. The evaluation will have summative and formative elements 
to it, but will primarily aim to be forward looking to understand better what CLEAR is achieving, 
whether its objectives are being met, and whether the approach being taken is appropriate in order 
to ensure effective performance for the remaining period of the initiative and beyond. 


The objectives of the evaluation are to: 


1. Highlight achievements, challenges and lessons to date as a basis for accountability to the 
funders and hosting institutions. 


2. Make recommendations for improvements in the design, management, governance and 
implementation of the CLEAR Initiative for the remaining period of implementation, and with 
a view to taking the initiative to scale. This includes identifying the most promising 
strategies and/or alternatives approaches for CLEAR’s success. 


3. Produce public good knowledge (lessons, approaches) as part of the evaluation to inform 
the fields of development evaluation, regional capacity building, institution building, and 
global initiatives. 


6. Evaluation Questions 
The evaluation will draw on the standard OECD DAC criteria to guide the selection of key 
evaluation questions, namely relevance, effectiveness, efficiency and sustainability?, 
Relevance will address the following key questions: 

1. Is there a genuine demand or need for CLEAR as per its value proposition? 


2. Is the theory of change for CLEAR relevant (original and revised) and is the results 
framework appropriate? Do the original assumptions still hold true? 


3. Does the design of CLEAR respond to effective and latent demands for development 
evaluation globally and in its respective regions from the various groups of stakeholders 
that it targets? 


4. How does CLEAR respond to demand in terms of systems and processes at the global and 
Centre levels (work plans, human resources, target setting etc)? 


5. Are the regional centres fit-for-purpose in terms of responding to the needs? 
6. Are the Secretariat’s and the Board's role and configuration appropriate to the needs? 


3 The evaluators will not be expected to look at impact at this mid-term point. 
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Effectiveness will address the following questions: 


1. Arethe Centres set-up to achieve results, and is there consistency and appropriateness in 
the results they aim to achieve (internally within their institutions) and across CLEAR? 


2. Are intended initial (emergent) results of CLEAR being achieved at global and regional 
levels? Is the Secretariat delivering value? 


3. Is there evidence of unintended effects of CLEAR positive or negative, e.g. undermining 
market 


4. What are the factors and mechanisms that support the delivery of results and those that 
hinder them? Are they context / Centre specific or CLEAR-wide? 


5. What influence has CLEAR achieved to date, in particular the extent to which critical 
stakeholders have been motivated and stimulated to engage in CLEAR, to change 
attitudes, behaviour, practices and systems in support of the objectives of CLEAR? 


6. To what extent is CLEAR is perceived as innovative by its stakeholders globally and 
regionally. How well has Secretariat carried out its role, and that of the Board? 


Efficiency will address the following questions: 


1. To what extent are the funds being used appropriately and generating value-for money both 
globally and within each regional centre (where VFM pertains to the extent to which issues 
of economy, efficiency and cost-effectiveness are embedded or not in working practices) 


2. Can the costs of delivery of similar services be benchmarked across each centre, given 
variances in the respective costs of doing business, geographies and other factors? If so, 
what are the findings? 


Sustainability will address the following questions: 


1. What is the viability of the CLEAR business case and the potential for sustainability beyond 
the life of donor financing? 


2. Which are the most promising strategies for success? 


3. What may need to be changed, and what further developments are required to take the 
Initiative to scale, and what the scaling-up options might be (such as more influence in 
existing regions and / or more centres / regions)? 


4. What level of commitment and interest exists within the Centres (staff and governing 
bodies) to carry CLEAR forward a) within their institutions and b) as an inter-Centre global 
initiative beyond existing external financing? 


7. Evaluation Approach and Scope 


The approach to evaluation will be guided by the questions posed and available data. As an 
evaluation at the mid-point in its cycle, the focus will not be on potential impacts, but focus heavily 
on the design features, the validity of the theory of change and the delivery of initial results. 


The evaluation will therefore be guided by a theory-based approach, where the theories that will 
inform this evaluation are drawn from the approaches that other, successful, global capacity- 
building initiatives have taken. The evaluators will thus be expected to present, in their evaluation 
design, evidence from comparative models of innovation, service delivery in the areas of training 
and technical assistance, leadership in development evaluation and building effective networks as 
the basis for assessing whether CLEAR is delivering. 





e-Pact 5 


CLEAR Midterm Evaluation — Annexes 





The theory-based analysis is mechanism-based, and will look for connections between cause-and- 
effect addressing not only the question of whether the programme is working against its own terms, 
but what it is about the programme that is or isn’t working, and why. It will also reveal typical 
patterns of development of such initiatives to see how CLEAR reflects these patterns — within each 
centre and as a global initiative. 


The evaluation will include a social network analysis (SNA) of CLEAR, to map and measure the 
relationships and flows between the CLEAR secretariat, the Centres and their clients/stakeholders 
in order to determine whether or not the reach of CLEAR to a range of stakeholders and clients 
across government, civil society and private sector is gradually increasing and strengthening over 
time*. The SNA will provide both a baseline for follow-up analysis at a later stage, and provide an 
indication of the reach of CLEAR at this stage. It is expected to guide both the Centres from an 
operational perspective, and the Secretariat / Board in terms of reach to-date. 


The type of methods to be used are expected to include network analysis®, stakeholder interviews, 
opinion leader surveys, interviews with service providers and innovative ways of seeking 
performance feedback such as crowdsourcing. Methods will be further developed by the evaluators 
in the inception phase. 


Scope 


The evaluation will cover all regional centres, the global secretariat, the governance structures — 
global Board (funders at present), regional Advisory Committees (RACs). It will seek to address the 
concerns of the Board and members of the governing bodies of the regional centres, the 
secretariat and of the centres themselves. The evaluation will also seek to provide performance 
and financial information required by funding agencies carrying out their periodic reviews of their 
support as a means to reduce the number of similar programme reviews and hence transaction 
costs for all concerned. 


The evaluation as a process should be independent and impartial but focus on learning, draw in all 
key stakeholders at the relevant stages of design, implementation and presentation of findings. 
This will facilitate a dual objective of providing information on how and where CLEAR should go, 
and facilitating the inter-centre relationship amongst those responsible for the strategic oversight of 
these institutions to see how they can work together in the future. 


8. Evaluation Team / Consultants 


The evaluation will be commissioned to external consultants by means of a Request for Proposals 
(RFP) process through DFID’s global evaluation framework agreement with pre-qualified suppliers. 
The evaluation will managed by the Board task force on the evaluation, and overseen by an 
evaluation committee composed of CLEAR Board representatives, representatives of the 


4 Social Network Analysis (SNA) is a technique to identify the relationships that an entity has (in this case 
CLEAR), with whom, and the flows between the nodes in the network. The nodes in the network are the 
people and groups while the links show relationships or flows between the nodes. SNA provides both a 
visual and a mathematical analysis of human relationships. To understand networks and their participants, 
we identify the location of actors in the network. Measuring the network location is finding the centrality of a 
node. These measures give us insight into the various roles and groupings in a network -- who are the 
connectors, mavens, leaders, bridges, isolates, where are the clusters and who is in them, who is in the core 
of the network, and who is on the periphery, who is missing in the target groups? 


5 To conduct an SNA involves interviews and surveys by phone and email with intended participants and 
clients of CLEAR in order to map who they are, where they are and who they are connected to, and the 
strength and frequency of those connections. The work proposed for the evaluation also includes surveys, 
interviews, presentation and discussion of the data with the CLEAR Centres, Secretariat and Board. 
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governing structures of the CLEAR implementing institutions. Procurement and process 
management of the consultants will be managed by DFID, Evaluation Department (John Murray) in 
collaboration with the evaluation committee. 


The consultants’ team will be composed of at least six members. These should include one 
person based in each region (Africa®, Asia, South America) led by two members (team leader and 
deputy) with an overall mandate to assess the global component, undertake the network analysis’ 
and to coordinate and guide the overall evaluation.. The expertise of the team should include 
appropriate academic qualifications (economics, social science, policy, or business), substantial 
experience in capacity building, monitoring, evaluation, innovation and strategic planning, network 
analysis and a proven track record of high quality multi-country studies. The team should have a 
good balance of expertise from the public and private sectors and from civil society. The majority of 
team members should be from developing countries (Asia, Africa and Latin America). 


In addition, the expected skills of a balance of creative and critical thinking; team work and 
collaboration; written and oral communication skills appropriate for undertaking evaluation 
fieldwork and reporting findings; interpersonal skills including ability to relate to people at all level; 
linguistic skills (written and oral) in English, French and Spanish (across the team), balance of 
gender, and a commitment to ethical evaluation practice and values consistent with the CLEAR 
Initiative are required. 


9. Outputs* and timeline 


The Evaluation Team will be expected to deliver the following outputs*, at the following intervals: 


Contract signed By November 7, 2013 


Conduct primary data collection, then attend and make a presentation” of November 18-22, 2013 
initial concept at the 2"¢ CLEAR Global Forum, being held in Mexico 


Draft Inception report*, including detailed evaluation approach, methodology, December 15, 2013 
work plan, communications plan and budget 


Field visits and data collection December 2013 - February, 
2014 

Presentation of Preliminary Findings (Power Point presentation) to March 3, 2014 

Evaluation Committee 

First draft evaluation report March 31, 2014 

Final Evaluation Report April 30, 2014 

Dissemination activities (TBD) May 2014 


* Additional outputs may be required, which will be discussed during the inception phase with the consultants. 


e This contract will commence in November 2013 for a period of 7 months expiring in May 
2014. The inception phase will be for a period of 1 month maximum. DFID reserves the 
right to extend the contract on a monthly basis, on the basis of continued need and 
availability of funding. 


e DFID reserves the option to scale back or scale up funding depending on performance and 
on-going need. 


6 It is expected that two consultants are needed for Africa, one to cover the Anglophone Centre and 
countries, the other for Francophone. 


7 If the team does not have the expertise / experience in network analysis, this component may be sub- 
contracted. 
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10.Roles and Responsibilities in the Evaluation 


Consultants / Evaluation Team will be responsible for: 


Engaging qualified evaluation practitioners to carry out the evaluation. 
Developing a detailed work plan and methodology. 


Conducting the evaluation in a way that supports capacity development with key 
stakeholders and facilitates learning. 


Delivering the learning and evaluation outputs in line with the agreed work plan and to a 
level of quality acceptable to the CLEAR Board. 


Managing the administrative and logistical requirements of the evaluation, including travel 
and field work. 


Presenting the progress of the evaluation to the Board Evaluation Committee, and the 
results of the Evaluation to the Board and CLEAR Centres staff. 


CLEAR Global Secretariat will be responsible for: 


Giving access to an existing Team Room site with all the essential CLEAR documents for 
the evaluation. The Secretariat will organize the documents in a way that is easily 
understood and accessible to the team. 


Providing inputs into and reviews of the draft report - as key stakeholders in the evaluation 
(in addition to the Centres). 


Providing guidance on appropriate scheduling of monitoring field visit itineraries, and 
providing letters of introduction for the consultant where appropriate. 


Providing feedback to the consultant on factual accuracy and utility of draft reports and 
other products. 


Providing letters of introduction for the evaluators where appropriate. 

Providing background information on CLEAR. 

Responding to the requests of the evaluators for additional information. 

Receiving and considering the recommendations of the reports and report on actions taken. 
Reporting to the Board on the actions proposed to act on the recommendations. 
Communicating and discussing the key findings with their respective stakeholders. 


Integrating recommendations into ongoing operational plans for CLEAR at global and 
regional levels. 


Regional Centres will be responsible for: 


Working collaboratively with the Board, Secretariat and the evaluators in the design of the 
Evaluation. 


Providing information and access to CLEAR activities and results. 


Supporting the consultants in providing names for interviews, contacts, facilitating 
interviews with their senior people, etc. 


Providing ongoing operational and programmatic information to the consultant. 


Providing administrative liaison with the consultants. 
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e Providing guidance on appropriate scheduling of monitoring field visit itineraries, and 
providing letters of introduction for the consultant where appropriate. 


e Facilitating field visits of the evaluation team including to CLEAR host institutions, public 
and civil society stakeholders in the region. 


e Reviewing the draft evaluation report. 


e Providing feedback to the consultant on factual accuracy and utility of draft reports and 
other products. 


e Participating in the presentation and discussion of the results of the evaluation to the 
CLEAR Board. 


e Receiving and considering the recommendations of the reports and report on actions taken. 
e Communicating and discussing the key findings with their respective RACs and partners. 
e Integrating recommendations into ongoing operational plans for respective CLEAR Centre. 


Evaluation Task Force of Board (a task force of a core of Board members responsible for leading 
on the design and coordination the evaluation) will be responsible for: 


e Key audience for receiving recommendations for consideration (some actions may have 
direct implications for the Board) and for approving the action plans pertaining to the 
findings and recommendations. 


e Drafting the Terms of Reference. 


e Establish and update the Evaluation Committee (including representatives of regional 
centres governing bodies). 


e Commissioning the evaluation. 


e Approve the Evaluation Team’s Inception Report (including work plan, budget and 
methodology) and subsequent outputs including the evaluation reports. 


e Overseeing the implementation of the Evaluation, including regular check-ins, reviewing 
reports, providing feedback, facilitating reporting of the evaluation at the CLEAR Forum. 


Evaluation Committee (CLEAR Board and representatives of Regional Centres governing bodies) 
will be responsible for: 


e Providing comments on the Terms of Reference for the Evaluation 

e Provide comments of draft products 

e Communicating the findings of the Evaluation to member funding agencies. 

e Overseeing the strategic implementation of recommendations for CLEAR. 
UK Department for International Development will be responsible for: 

e Leading the procurement process with the guidance of the evaluation committee 

e Financing the evaluation on behalf of the Board 
Be the primary point of contact for the evaluators as pertaining to contractual issues (the project 
leader will be Mr. John Murrray (j-murray@dfid.gov.uk), Evaluation Business Manager of DFID’s 
Evaluation Department. 
11.Indicative Budget 


A budget of up to £350,000 has been allocated to this work. 
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12.Constraints, Dependencies, Risks and Challenges 


The evaluation contract should be issued by the end of November 2013 to allow the evaluators to 
make active use of the second CLEAR global forum to be held in Mexico City between 18-22 
November 2013 at which all CLEAR centres, board members and other key stakeholders will be 
present. 


The risks to the evaluation are primarily threefold: 


First, the CLEAR theory of change does not, currently, effectively elaborate the critical 
assumptions and cause-and-effect relationships between the supply through centres and the aim 
of improved use of evidence by decision makers. These assumptions need to be identified, 
carefully though through in each context and for CLEAR as a whole, and tested through the 
evaluation is if it to effectively identify how best to move forward. 


Second, some of the centres, notably the Francophone Africa Centre have only recently begun 
operating, and thus the strategies are still being formulated and few results will have yet been 
achieved. This will require the evaluators to be flexible in terms of the criteria they employ to 
assess performance / progress. 


Third, CLEAR itself is has a broad geographical scale and ambition, and the views of the member 
institutions, the governing bodies of these institutions, the Board, the regional advisory committees 
and other stakeholders are likely to vary considerably in terms of current and future expectations. 
The evaluators may face challenges in addressing and reflecting these various perspectives ina 
coherent manner within the compressed timeframe available. 


13. Quality Control 


The evaluators will be required to follow standard ethical practice in evaluation.® In terms of the 
quality of the methodology and outputs, the CLEAR Initiative follows the OECD-DAC Evaluation 
Standards and reference should be made to the IEG/DAC Network sourcebook for evaluating 
global and regional partnership programmes.* Quality assurance will be carried out by the 
Evaluation Committee of the Board, guided, as appropriate, by independent evaluation, innovation 
and capacity building experts who will be engaged informally during the process. This will not be 
an official reference group, which is not deemed necessary for this mid-term evaluation. 


14.Reporting arrangements 


The Consultants will report to the Chair of the Evaluation Task Force through the Project 
Leader/Manager, John Murray (j-murray@dfid.gov.uk) 


$ Such as DFID's ethics principles for research and evaluation. 
9 


http://www.bing.com/search?q=IEG+sourcebook+for+evaluating+global+and+regional+partnership+program 
mes&src=ie9tr 





e-Pact 10 


CLEAR Midterm Evaluation — Annexes 





Annex B Evaluation Methodology and Approach 


B.1 Evaluation Approach 


In keeping with the TOR, the evaluation adopted a theory-driven approach in order to provide 
information not only on CLEAR programme performance but also to help explain how and why the 
programme is or is not achieving intended results. Thus the Evaluation Team considered both the 
explicit and implicit assumptions of the Theory of Change and Results Framework when reviewing 
progress towards the outputs and outcomes stated in these frameworks, taking into account that 
CLEAR was designed as an innovative initiative that is still evolving.'° In addition, the team 
developed a draft alternative Theory of Change (see Annex E) to illustrate the types of 
assumptions that are not yet captured by the CLEAR ToC. This approach was intended to help 
serve both the accountability and programme improvement expectations of CLEAR stakeholders. 


Comparing CLEAR to other relevant actors and programmes — Part of applying a theory- 
based approach involved exploring if or how CLEAR contributions to results can be distinguished 
from the contributions of other actors who work in the same or similar areas or that share certain 
attributes/objectives with CLEAR. Recognising that there are differences among Regional Centres, 
the evaluation — to the extent possible - compared each Regional Centre to other relevant key 
actors in the same region in terms of reputation, cost and unique services. Although it was not part 
of the TOR, stakeholders were interested in how the CLEAR organisational arrangements compare 
to those of other global partnership programmes. For this purpose, the Evaluation Team reviewed 
the governance structures of a few similar global programmes including the Consultative Group on 
International Agricultural Research (CGIAR), the Consultative Group to Assist the Poor (CGAP), 
and the Global Fund to Fight AIDS, Tuberculosis and Malaria. 


Principles of Inclusion and Equality — The evaluation made efforts to ensure that the 
stakeholders consulted represented diverse perspectives (based on gender, ethnicity, geographic 
locations, and other locally relevant criteria) and solicited the views of both rights holders and duty 
bearers. 


B.2 Evaluation Framework 


The evaluation Inception Report outlined the following three components of the evaluation: 
e CLEAR Summary of Progress at Mid-Term 
e Analysis of CLEAR Programme Level Performance (both global and regional) 
e Analysis of CLEAR Organisational Arrangements (factors affecting performance) 


In its feedback on the Inception Report, the Evaluation Task Force (ETF) had requested that the 
evaluation report focus primarily on issues concerning the future of the programme. In the first draft 
of the evaluation report, progress at midterm was therefore presented in a separate Volume II. 


In its comments on the draft evaluation report the (broader) Evaluation Committee requested that 
i) more information on progress made at midterm be brought into the body of the main report, and 
ii) that the report be restructured to provide more information on progress made by each centre 
and less emphasis on CLEAR governance and organisational arrangements. These requests are 
reflected in the structure of the revised evaluation report, and the omission of the separate volume 
summarising CLEAR progress against its midterm indicators. 


10 See Inception Report (February 2014) for details on testing assumptions. 
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Evaluation Matrix 


Based on the TOR, and in consultation with the ETF, the Evaluation Team developed a matrix of 
evaluation questions to guide data collection, data analysis and reporting writing. The evaluation 
questions cover the evaluation criteria of relevance, effectiveness, efficiency, and sustainability. 
For each question, sub-questions were developed to define the issues and the basis for judgment 
with greater precision. Indicators were developed for each sub-question. The evaluation questions 
are presented in Annex C and the data collection matrix in Annex D. 


The evaluation questions and criteria were used to assess the performance of CLEAR at global, 
regional and national levels. They were also used to develop standardised interview protocols and 
survey instruments to ensure consistent collection of data, to allow for triangulation of data from 
different sources, and to assist in the final synthesis of findings and recommendations. 


B.3 Evaluation Management and Process 


The evaluation was commissioned by DfID and managed by the Evaluation Task Force (ETF). 
The ETF established an Evaluation Committee’? and worked with the Universalia evaluation team 
throughout the evaluation (providing guidance, reviewing reports and presentations, providing 
feedback, resolving challenges). 


The evaluation process consisted of four phases as described in the sections that follow. 


Data collection Analysis and Dissemination 


os and field visits reporting and follow-up 





B.3.1 Phase | — Inception 


The Inception Phase was intended to fine-tune and clarify the evaluation purpose, objectives and 
methodology. 


During the Inception Phase the Evaluation Team met with the CLEAR Board to clarify the 
evaluation purpose and objectives and review and revise the methodology and composition of the 
team (e.g. having Southern representation on the Evaluation Advisory Committee and changing 
one of the evaluation team members). 


During this phase, the Evaluation Team interviewed more than 40 CLEAR stakeholders (including 
CLEAR Board members, Regional Centre directors and staff, representatives of host institutions, 
as well as Secretariat staff) to obtain their views on how the evaluation could add most value to 
CLEAR and their organisations. The team attended the global forum in Mexico in November 2013 
where it interviewed stakeholders and participated in a CLEAR workshop on financial 
sustainability. Stakeholders were consulted either in Mexico during the CLEAR Global Forum 
(November 2013) or by telephone/Skype from 2-11 December 2013. 





11 The ETF, which is responsible for leading on the design and coordination of the evaluation, consists of 
four members of the CLEAR Board: representatives from DFID (Chair). AusAid, the Rockefeller Foundation, 
and SIDA. The designated SIDA representative was not able to participate. 


12 The committee includes: representatives from the CLEAR Board (the Evaluation Task Force, the World 
Bank, AFDB, IADB); representatives of donor agencies that support CLEAR but that are not on the Board 
(ADB, SDC); and representatives of Regional Centre governing bodies, namely from Witwatersrand 
University (South Africa), CESAG (Senegal), IFMR (India), and CIDE (Mexico). 
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The Evaluation Team reviewed a number of key CLEAR documents to identify the types of 
information available for the evaluation, and also consulted external experts to obtain their insights 
on the design of the network analysis component of the evaluation. 


In coordination with the ETF and with the assistance of the Secretariat, initial contact was made 
with the heads of five Regional Centres’ to confirm the timing of the field visits and to outline 
expectations and support requirements during the data collection phase. 


The Evaluation Team developed a working draft of key assumptions associated with the Theory of 
Change and discussed these with representatives of the CLEAR Board and Secretariat in order to 
identify an initial list of issues that required further exploration. 


A draft Inception Report was submitted to the ETF in December 2013; the final version was 
approved in February 2014. 


B.3.2 Phase Il - Data Collection 


B.3.2.1 Overview 


Data collection took place between November 2013 and May 2014. The Evaluation Team, 
supported by regional consultants, collected data to: inform the organisational assessment of each 
Regional Centre; obtain information required to assess the performance of the CLEAR programme 
in each region; inform the network analysis. 


The Evaluation Team used a mix of quantitative and qualitative methods as described below. 
B.3.2.2 Document Review 


During the course of the evaluation, CLEAR stakeholders provided the Evaluation Team with input 
on relevant documents. The Evaluation Team carried out a second round of in-depth document 
review in order to generate information to address the key evaluation questions. Relevant 
information was coded and organised by criteria and sub questions and/or by other key foci that 
emerged during the course of the evaluation. This facilitated the sorting, analysis and triangulation 
of data to inform analysis and report writing. The Evaluation Team reviewed various types of 
documents, including minutes of Board meetings made available by the EFT/CLEAR Secretariat / 
CLEAR Board, strategy documents, reports commissioned by CLEAR since establishment, DfID 
reports, and documents specific to the Regional Centres, etc. The list of documents reviewed is 
presented in Annex G. 


B.3.2.3 Interviews with Key Stakeholders 


The Evaluation Team consulted 273 stakeholders to obtain their perspectives on the key 
evaluation issues. The list of stakeholders consulted is presented in Annex F. The table below 
illustrates the types of stakeholders consulted and their relationship to CLEAR. 


Types of organisations that consulted stakeholders represented 


Level/ Academic/ NGO/ Government Government Individual Donor/ Other Total 
Centre Research (02910) (national) (sub-national) (consultant) Development 
organisation 
2 
AA 5 










Global 


FA 
EA 10 4 4 2 
LA 22 1 9 il 
SA 


18 Since the Regional Centre in Brazil is very new, the Evaluation Team consulted stakeholders by 
telephone. 
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Relationships of interviewed stakeholders with CLEAR 


Level/ | CLEAR | Thought | CLEAR | Host | Actual | Actual | Actual | Potential | Potential CLEAR Other | Total 
Centre | donors leader staff & | inst. | client | partner | partner | partner client Competitor 
faculty & 
client 
Global 11 7 Ee 
AA 10 Y T 16 1 1 6 


SA iy 2 11 3 4 1 
Dies D Ea Sen eS A AAA RS 
During the inception phase CLEAR Regional Centres were asked to suggest contacts in key civil 
society and private sector organisations engaged in M&E capacity building in the region who could 

be interviewed for the evaluation. The centres varied in their ability to generate names and 
contacts. This ability was not necessarily linked to the scope of existing partnerships or contacts 
that the respective centre had established. From the outset and throughout the assignment, the 
evaluation team actively sought input from all consulted stakeholders to identify additional key 
persons to speak with, including actors who had not received CLEAR services previously. This 
increased the potential list of respondents and we attempted to interview everyone identified (in 
some cases this included repeated requests and reminders); however, some never replied. 
Following feedback received on the first draft of the report, the evaluation team asked the CLEAR 
Board to suggest additional stakeholders (in particular NGOs/CSOs) at global or regional/national 
levels, but did not receive any such suggestions. 


loba 5 
2 ESS 
FA 2 14 16 30 11 1 2 ies 
EA 5 6 6 3 2 0 3 3 Esa] 
LA 8 9 8 9 12 4 4 1 fen 
5 4871] 
[52731] 


The team carried out interviews in person during the regional site visits and in Washington DC and 
also through videoconference or phone/Skype. All interviews were guided by protocols organised 
around the major evaluation questions (see sample protocols in Annex |). 


Given that this is a midterm evaluation, our focus was on learning about the types of benefits and 
challenges/ issues that CLEAR staff, clients and partners are raising, rather than on measuring 
impact. We therefore used semi-structured interviews rather than a written survey. This allowed for 
qualitative discussions with stakeholders but did not yield quantifiable results. However, in order to 
validate some mixed views on the utility of the impact evaluation courses to participants in 
Anglophone Africa, the Evaluation Team, following the site visit to the AA Centre, launched a short 
email survey to gather the views of individuals who attended IE courses in Uganda and South 
Africa in 2013 and 2014 (see summary report in Annex P). 


In the report when we refer to ‘several stakeholders mentioned x’ this means that at least three 
individuals in a region or at global level raised a particular issue or had a similar point of view, 
independently from one another. We took this as an indication that the issue or viewpoint was 
important enough issue to be brought up for CLEAR consideration. Where applicable we indicate if 
a particular view was expressed only in a certain region, or only by certain types of stakeholders 
(e.g. CLEAR clients but not staff). When we say ‘most’ stakeholders in a region said x’, this 
indicates that the issue or point of view was consistently confirmed by everyone we talked to — i.e. 
there were no distinct differing viewpoints. 


B.3.2.4 Site Visits to CLEAR Regional Centres and Affiliate Centres 


The Evaluation Team, supported by regional consultants, conducted visits to five Regional Centres 
(Anglophone Africa, Francophone Africa, East Asia, South Asia, Latin America and the Caribbean) 
and collected data from CLEAR staff and stakeholders (clients, collaborators, competitors, and 
thought leaders) to: inform the organisational assessment of each Centre; obtain information 
required to assess the performance of the CLEAR programme in each region; and inform the 
network analysis. 


The visits were conducted by one international consultant and/or one regional consultant, as 
shown in the sidebar. 
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In addition, the team conducted field visits to affiliate centres in Ghana, Kenya, and Burkina Faso 
to assess the organisational arrangements the affiliates have with CLEAR and, where feasible, to 
review selected CLEAR-funded initiatives in that country. Stakeholders connected to the affiliate 
Centre in Pakistan were consulted via telephone and Skype.** Furthermore, in the case of the Latin 
America Regional Centre which has no 

affiliate Centre, a visit was conducted to Anglophone Africa 

Peru to consult with CLEAR clients and 


South Africa - Geraldine Cooney and Adeboye Adeyemo 
collaborators. 


, , NN , Kenya and Ghana - Adeboye Adeyemo 
During the site visits, the Evaluation Team 


conducted interviews and small group Francophone Africa 


meetings with a broad range of informants. Senegal - Heather Baser and Ahmed Bencheikh 
Information gathered during each site visit Burkina A AhmediBencheikh 

was consolidated, analysed, and used to i 

prepare tables summarising: the East Asia 

organisational performance of each China - Sadie Yang 

Regional Centre, the CLEAR programme South Asia 


progress to date in each region, and the | 

viability of each centre. At the end of (or India - Anette Wenderoth 
immediately following) each regional visit, Latin America 

the Evaluation Team shared highlights with 
Regional Centre staff (and others as 
deemed appropriate) using a PowerPoint Peru — Rosa Flores 

presentation summarising preliminary 

findings, conclusions and emerging recommendations. These findings, combined with those 
emanating from the document review, were later aggregated to develop crosscutting findings 
related to the key evaluation issues and to develop a set of recommendations. 


B.3.2.5 Site visit to CLEAR Secretariat 


The Evaluation Team Leader, supported by other core team members as required, collected data 
during site visits to Washington, DC and also through telephone/Skype and email to inform the 
reviews of the CLEAR Secretariat and of the network. 


Mexico - Elisabetta Micaro and Rosa Flores 


B.3.3 Phase Ill - Analysis and Reporting 


B.3.3.1 Analysis 


The questions in the evaluation matrix were used to structure data analysis. The matrix provided a 
template for the Evaluation Team to formulate findings on the basis of the information collected at 
three levels: 


e At the level of indicators, taking into account all information that had been collected for each 
indicator. 


e At the level of the sub-questions, across all indicators associated with the respective sub- 
question. 


e At the level of the evaluation question, aggregating information collected for each sub- 
question. 


The following methods of data analysis were employed to make evaluative judgments against the 
agreed upon basis for assessment, i.e. the CLEAR Theory of Change and Results Framework. 


14 This approach was suggested by the affiliate Centre in Pakistan and derived from the consideration that 
the number of stakeholders to be consulted in Pakistan is relatively limited, and that the respective 
individuals are located in different parts of the country, which would limit the usefulness of a visit to Lahore. 
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Descriptive analysis was used to understand the contexts in which CLEAR works and to describe 
its portfolio of interventions. Descriptive analysis was used as a first step, before moving on to 
more interpretative approaches. 


Content analysis constituted the core of the qualitative analysis. Documents and interview notes 
were analysed to identify common trends, themes, and patterns for each of the key units of 
analysis. Content analysis was also used to flag diverging views and opposing trends that might 
require further data collection. Emerging issues and trends constituted the raw material for crafting 
preliminary observations that were subsequently refined to feed into the draft and final evaluation 
reports. 


Quantitative analysis was used to interpret quantitative data. It was used principally to assess 
CLEAR's use of resources, and to quantitatively analyse different characteristics of the 
interventions portfolio and the results or the ONA as categorised by geographic, thematic, or other 
criteria. 


Comparative analysis was used to examine findings across regions, countries, themes, planned 
CLEAR contributions, CLEAR’s reputation, and other criteria deemed important. This type of 
analysis was used throughout the process to examine information and data from stakeholder 
consultations and document/file, the ONA, and literature review. 


In addition, elements of contribution analysis were used. At midterm, the types of results that 
CLEAR has contributed to are located at the level of outputs, i.e. they can be directly linked to 
specific CLEAR (centre or global) activities. Given that there was no significant evidence of 
progress towards outcomes at this point, it was neither necessary nor possible to apply 
contribution analysis in its full sense (i.e. trying to establish CLEAR contributions to outcomes). The 
evaluation team therefore focused on employing elements of contribution analysis, in particular a 
review of the existing Theory of Change and the development of a draft alternative ToC (which was 
deliberately set at a generic level to demonstrate the types of assumptions that have not yet been 
fully articulated). CLEAR might use this as a starting point for further discussion of the ToC. 


Triangulation of Methods and Data Sources 


The mixed methods purposefully influenced the analytical process in that they provided 
opportunities for triangulation through the convergence and overlapping of different methods. The 
Evaluation Team also triangulated data from different sources to ensure the reliability of 
information and to increase the quality and credibility of the evaluation findings and conclusions. 
Specifically the Evaluation Team: 


e Collected information from multiple data sources for each evaluation question, including a 
broad variety of stakeholders and documented data (primary and secondary sources). 


e Used a mix of data collection methods (both quantitative and qualitative) at all levels of 
analysis (global, regional, and national). 


B.3.3.2 Reporting 
The evaluation produced several kinds of reports as described below. 
Presentation and Validation of Preliminary Findings 


Towards the end of the data collection phase, the Evaluation Team identified and synthesised the 
emerging findings, issues, trends, and opportunities to respond to the key questions for the overall 
evaluation. These were summarised in a PowerPoint slide presentation. The Team Leader and 
selected members of the team shared this presentation via videoconference with the Evaluation 
Task Force and other stakeholders. This exercise was used to validate emerging findings, discuss 
and resolve potential issues, and inform subsequent data collection if and as necessary. 
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First Draft, Second Draft and Final Evaluation Reports 


Following the completion of data collection and analysis, the Evaluation Team prepared a first draft 
evaluation report, on which it received written feedback from members of the Evaluation 
Committee. Subsequently, the evaluation team attended a working session with CLEAR Board 
members in Washington D.C. (July 2014). The meeting was used to ensure shared agreement on 
key messages deriving from, or missing in the draft evaluation report, and to ensure a shared 
understanding of Board members’ needs and expectations as regards the structure and foci of the 
report. 


Following this session, and taking into account feedback received from the Evaluation Committee, 
the report was revised. The evaluation team will seek one round of consolidated feedback from the 
Evaluation Committee to prepare the Final Evaluation Report. 


B.3.4 Phase IV — Communication and Dissemination of Evaluation Results 


All three purposes of the midterm evaluation (accountability to funders, learning for improvement, 
and public knowledge improvement) require effective communication and dissemination of 
evaluation results to difference audiences. Table B.1 outlines the activities proposed by the 
Evaluation Team to communicate and disseminate information on the evaluation to key 
stakeholder groups. The continued appropriateness of these suggestions will be revisited with the 
ETF following the submission of the second draft evaluation report; dissemination products will be 
revised as required to reflect evolving stakeholder needs. 


In addition to these products/activities the team suggests that the final evaluation report be made 
available on the CLEAR website once it has been approved by the Board. 


Table B.1 Draft Communication and Dissemination Plan 


Audience/ Suggested communication products/ dissemination events Suggested timing 
Target Group 


CLEAR Board Presentation and discussion of key evaluation findings and During Board 
recommendations meeting in early July 
2014 
CLEAR Up to three webinars facilitated by 1-2 Evaluation Team Members Autumn 2014. Dates 
Regional for CLEAR Regional Centre staff and key clients/collaborators to be discussed with 
Centres (invited by the respective Centre of Affiliate Centre) to share and the Regional 
discuss key evaluation findings and recommendations. Each Centres. 


webinar would last approximately 1.5 hours. If desired by the 
Regional Centres one webinar each could be conducted in 
English, French, and Spanish respectively. 


Interested Up to three written Learning Briefs based on the final evaluation Autumn 2014. Dates 

public at large report that can be made available to the broader public through to be discussed with 
the CLEAR website, the websites of the Regional Centres, and/or the Evaluation Task 
other channels to be determined by the ETF. Each brief will Force. 


summarise key evaluation findings and observations around a key 
theme or question on 1-3 pages. The themes/questions to be 
addressed will be discussed with the ETF once the final 
Evaluation Report has been submitted. 


B.3.5 Limitations and mitigation strategies 


Table B.2 presents some limitations that were encountered during the evaluation that adversely 
affected the evaluation results, as well as related mitigation strategies employed by the Evaluation 
Team where applicable. 
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Table B.2 


Limitations and Mitigation Strategies 


The evaluation TOR and, in consequence, the 
resulting evaluation methodology were not fully suited 
to address the experimental nature of CLEAR. The 
evaluation used traditional evaluation criteria, 
categories and tools, such as the results framework to 
assess an evolving initiative at a single point in time, 
rather than conducting an ongoing (developmental) 
evaluation. This was identified as a shortcoming in the 
evaluation design during the meeting with the CLEAR 
Board in July 2014. 


The original evaluation design included an 
organisational network analysis (ONA) to map and 
measure knowledge sharing between and among key 
CLEAR stakeholders at global, national and regional 
levels. 

The conduct of the ONA encountered significant 
challenges. By July 2014 there were several delays 
and inconsistencies in completion of the ONA survey 
(including insufficient participation by three of the 
regions in which CLEAR operates) which limited the 
significance and utility of the information generated for 
the evaluation. 


The Evaluation Terms of Reference had indicated an 
interest in accessing information on how the cost of 
CLEAR centre services compared to those of other 
capacity building providers in the respective region. 
This proved difficult, starting with the fact that many of 
the services provided by CLEAR centres (e.g. tailored 
technical assistance to specific clients) are not also 
provided by other service providers, and are difficult to 
assign a specific value/amount to, as they can span 
over longer periods of time. Second, it proved similarly 
difficult to obtain reliable and comparable data at least 
on the cost of training activities (as the type of activity 
most frequently provided by other suppliers, too). 


For the revised evaluation report the evaluation 
team analysed available data and adjusted the 
findings to better reflect the experimental nature of 
CLEAR. 

While this was a limitation for the mid-term 
evaluation of CLEAR, it yielded some valuable 
lessons learned and alternatives on how 
stakeholders might wish to approach future 
evaluations of CLEAR or other experimental 
initiatives. 

The evaluation team sent out regular biweekly 
reminder emails to invitees based on level of 
survey response. In addition, targeted follow up 
emails were sent out to contacted/invited 
respondents in each region between 15 March 
and 2 May 2014. These measures did, however, 
not significantly enhance ONA response rates. 

In order to respect evaluation reporting timelines, 
in July 2014 the CLEAR Board agreed with the 
evaluation team’s proposal that: 

1) The ONA would be removed as a line of 
evidence for the evaluation report 

2) The ONA survey would be reactivated with the 
active support of CLEAR RCs as required to 
secure a minimum number of responses in each 
region. It is anticipated that the resulting analysis 
will be completed towards the end of 2014 and 
available for CLEAR as a standalone report. 


While the removal of this line of evidence reduced 
triangulation opportunities (particularly about the 
breadth and scope of each Regional Centre’s 
linkages to different stakeholder groups in each 
region, and to one another), this was to some 
extent mitigated by the large numbers of persons 
consulted in each region as well as the document 
review. Moreover, since the ONA will be 
completed, the results of the planned extension 
will be valuable baseline information that can be 
used to inform future reviews and evaluations of 
CLEAR's performance. 


Despite related efforts, the evaluation team was 
able to elicit only a few examples from some of 
the reviewed regions, some of which lacked 
specificity (e.g. on the exact scope of services 
covered by the quoted cost). The evaluation team 
therefore decided not to include this line of 
evidence in the report, as the available data did 
not permit any kind of triangulation and did not 
suffice to formulate related findings. 
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Annex C Evaluation Matrix (Questions) 


Part A - Summary of CLEAR Progress at Mid-Term 





1. Global Level 1.1 Establishment 1.1.1 What have been the respective Includes description of Secretariat roles and 
of CLEAR processes for establishing the CLEAR responsibilities in view of RCs and Board 
Secretariat and Secretariat and Board? 
Board 1.1.2 What types and amounts of resources 


(e.g. financial, human, infrastructure) has the 
Secretariat had access to as of December 2013? 


1.1.3 What types of managerial and 
administrative structures and processes has the 
Secretariat established (e.g. for M&E of initiative 
activities; administration)? What 
processes/structures have been established for 


the Board? 
1.2 Global level 1.2.1 What characterizes the portfolio of Focus on global programming activities. 
activities and activities conducted by CLEAR at the global level 
outputs to date? In terms of: 

a) Number and types of activities/outputs 


(e.g.: Knowledge exchange/sharing; Advocacy, 
Promotion; Knowledge Resources, etc.) 


b) Key topics addressed (if/as applicable) 


Cc) Proportion of funds used for different 
types of activities. 


2. Regional Level 2.1 Activities 2.1.1 What has been the process and timeline 
leading to the for selecting and establishing the Regional 
identification of Centres (and sub-centres??). 

Host Institutions 
and the 
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Part A - Summary of CLEAR Progress at Mid-Term 


establishment of 
Regional 
Centres! 


2.2 Regional 
Centre 
activities/outputs 


2.1.2 What types and/or amounts of resources 
(e.g. financial, human, infrastructure.) has each 
Regional Centre (had) access to as of December 
2013? 

2.1.3 What types of managerial and 
administrative structures and processes has 
each RC established (e.g. for M&E of centre 
activities; administration) 


2.2.1 What characterizes the portfolio of Whether and to what level of detail we can describe all 
activities and the allocation of resources dimensions will depend on the types of information/data 
conducted by the Regional Centres to date to available from the different RCs. 

support client capacity development? In terms 

of: 


a) Types of activities/outputs'® 


b) Geographic reach of activities (countries in 
region) 


c) Categories of change agents 
targeted/reached (Parliament, Ministry, 
Government Agency, Civil Society, Academia, 
Other) 


15 The midterm evaluation will focus on the five Regional Centres that had been established by 2012 and that were operational at the time of the 
evaluation, i.e. the RCs for Anglophone Africa, South Asia, East Asia, Francophone Africa, and Latin America and the Caribbean (LAC). The MTE will 
not include the recently created Centre in Brazil. 


16 Types of M&E capacity development activities outlined in the “Building Blocks of CLEAR’s Capacity Development Strategy” 
(http://www.theclearinitiative.org/ECD-CLEAR-ChangeAgents-ICOs-CD-Activities 4October%202013.pdf) are: training (various types); knowledge 
exchange/sharing; advocacy/promotion; leadership development; technical assistance and Organisational Capacity Building; Evaluations, 
Assessments, Project Advisory; Diagnostics; Knowledge Resources; Networks/Communities of Practice; Grants, competitions, awards; Other. The 
evaluation will also identify which activities were offered at a fee for service basis, and which one were free. 
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Part A - Summary of CLEAR Progress at Mid-Term 


d) Levels of change agents targeted/reached e topics) with those outlined in relevant planning 
(Executive, Managerial, Technical/professional, documents 
Other) e Stakeholder views on relevance and effectiveness 


e) Level of capacities primarily targeted by each eat pete UE activities ana poe dde 
activity (individual, organisational, system) e Stakeholder views on factors (internal and external) 


that have affected success of activities. 
f) Key topics addressed 


9) Distribution of funds among types of 
activities/categories of change agents/level of 
capacities targeted. 


3. Effectiveness 3.1 Global e To what extent have actual global e Comparison of actual activities/outputs (types, 
Programming programming activities been congruent with 
CLEAR’s strategic documents and 
frameworks? 


e What has been the degree of satisfaction of 
stakeholders for the various global 
activities? Why? 

e What common (process) factors led to 
success of individual activities? What has 
been learned on what did not work? 


3.2 Regional e To what extent have actual programming e Comparison of actual activities with 
Programming activities of the RCs been congruent with targets/objectives outlined in relevant planning 
their respective strategic plans and documents of each RC e.g. as regards: 
objectives? e Range of M&E topics and methodologies offered to 
e What types of benefits have CLEAR clients date 
derived from its services to date?17 e Range and distribution of modalities of capacity 
e What common (process) factors led to development (e.g. training, technical assistance) 
success of individual activities? What has offered by each RC 
been learned on what did not work? e Range and distribution of levels of change agents 


addressed by different activities 
e Range/diversity of M&E topics and methodologies 


17 Intermediate capacity development outcomes as per “Building Blocks of CLEAR’s Capacity Development Strategy” (http://www.theclearinitiative.org/ECD-CLEAR- 
ChangeAgents-ICOs-CD-Activities 4October%202013.pdf): Raised awareness, enhanced skills/knowledge; improved consensus/teamwork; enhanced networks; 
strengthened coalitions; new implementation know-how. 
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4. Relevance 


4.1 Context and 
Rationale 


4.2 Principles of 
inclusion and 
equality 





To what extent does CLEAR constitute an 
appropriate response to the national, 
regional and global trends underpinning its 
creation? Is it keeping up with evolving 
trends? 


To what extent are the rationale, logic, 
expected results and assumptions for the 
CLEAR programme clear, appropriate and 
commonly understood by CLEAR 
stakeholders? 


To what extent does the CLEAR design 
build upon and reflect (emerging) good 
practices in and current thinking on (M&E) 
capacity development, global networks, and 
global partnership programmes? 


To what extent do CLEAR objectives and 
services take issues of inclusion and 
equality into account? 


addressed 


Evidence of client satisfaction with type and quality 
of CLEAR services. 


Types of benefits that consulted RC clients note to 
have derived from RC activities 


Reasons given by consulted RC staff and clients for 
success (or lack thereof) of different activities. 


Network Analysis findings regarding current 
characteristics of the CLEAR network (e.g. in terms 
of reach and roles of different network 
members/participants in view of information 
exchange/use) 


Alignment of CLEAR response and global, regional 
and national trends. 


Extent to which key concepts inherent in the CLEAR 
design (including capacity development, innovation, 
demand-led, sustainability and so forth) are 
commonly understood by CLEAR stakeholders 
Extent to which key assumptions underlying 
CLEAR’s theory of change18 are shared by CLEAR 
stakeholders 

Extent to which the CLEAR design accommodates 
the risks inherent in innovative programming. 
Extent to which CLEAR strategic/planning 
documents make explicit reference to current 
concepts/thinking on (M&E) capacity development, 
global networks, and/or global partnerships. 

Extent to which CLEAR Secretariat staff, Board 
members, RC staff, and stakeholders see CLEAR 
as a reflection of current thinking in these areas. 


Evidence of CLEAR objectives, being reflective of 
the aim to foster inclusion and/or equality. 


Extent to which principles of inclusion and/or 


18 E.g. regarding the role of M&E and for evidence-based decision making and its link to improved development results 
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Part A - Summary of CLEAR Progress at Mid-Term 


4.3 Targeted 
regions/countries 
and change 
agents 


4.4 Donors 


4.5 Host 
institutions 


4.6 
Complementarities 
with existing M&E 
Capacity 
Development 
initiatives 


To what extent are CLEAR objectives and 
services congruent with regional and 
country needs and priorities? 


To what extent are CLEAR objectives and 
services congruent with targeted change 
agent needs and priorities? 


In what ways have CLEAR objectives and 
services taken the specific and changing 
contexts of different regions (political, 
economic, cultures as well as supply and 
demand for M&E) into account? 


To what extent are CLEAR objectives 
congruent with existing and potential 
donors’ priorities? 


To what extent are CLEAR Centers’ 
objectives and services congruent with host 
institution priorities? 


To what extent do CLEAR services and 
features complement or duplicate those 
services provided by other comparable M&E 
capacity development suppliers (global, 
regional, national) 


To what extent do CLEAR activities 
influence (positively or negatively) the work 
of other suppliers of M&E capacity 
development? 


equality have influenced decision making on 
capacity development activities and target groups. 


Alignment of CLEAR objectives and services with 
consensually identified/articulated regional and 
national needs and priorities. 


Alignment of CLEAR objectives and services with 
the needs of consulted stakeholders (individuals 
and representatives of organisations/institutions). 


Evidence of CLEAR objectives, services, and 
delivery modalities being reflective of and tailored to 
their respective (political, socio-economical) 
contexts, including on-going change processes 
characterizing these contexts. 


Reasons given by different categories of change 
agents/clients for (not) valuing CLEAR’s objectives 
and services. 


Types of CLEAR services/features most valued by 
current donors and reasons for doing so. 


Reasons why potential donors have not (yet) 
participated in CLEAR. 


Alignment with host institution priorities. 


Extent to which host institutions continue to 
consider their original reasons that led them to host 
the RC as valid. 


Number of instances and examples where CLEAR 
services complement or duplicate services available 
at global, regional, national 

Number and types of characteristics that distinguish 
CLEAR services from other similar services (e.g. in 
terms of quality, pricing, innovation character) 


Types of influences of CLEAR activities on local or 
regional markets 

Extent to which CLEAR stakeholders feel that 
CLEAR has a unique selling proposition or niche 
that differentiates it from other initiatives or 
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organizations. 


5. Outcomes and 5.1 Global e ls there early evidence that CLEAR is e Examples of Regional Centres sharing knowledge 
Sustainability Learning positioned to contribute to: and expertise through a global network 
Outcomes e Developing and sharing global knowledge? e Evidence Regional Centre directors and staff report 
Facilitating peer learning through the that they have been able to apply knowledge gained 
CLEAR network? from other Regional Centres through the CLEAR 
network 


nee Re tes (positive or negative) e Evidence of consulted CLEAR clients and 
effects emerging’ stakeholders and M&E experts recognize CLEAR 


e What factors support or hinder CLEAR's as a (potential) source of excellence and innovation 
positioning? Why? in M&E 


Types of unintended effects. 


Types of factors (internal and external) that have 
furthered or hindered CLEAR’s positioning. 


5.2 Regional e For each of the Centres, is there early e Evidence of CLEAR RC clients/stakeholders 
Learning evidence that CLEAR is positioned to reporting of (likely) progress towards strengthened 
Outcomes contribute to : organisational capacities due to CLEAR activities 
e Strengthening organisational capacity to and outputs. 
produce and use evidence in each Region e Evidence of (beginning) positive changes in the 
(i.e. addressing supply of and demand for available professional expertise for M&E in each 
evidence)? region. 
e Developing a critical mass of M&E e Evidence of increased innovation and diversity of 
professional expertise in each Region? M&E approaches in each region. 
e Innovation in and diversity of M&E e Types of unintended effects 
approaches/methods in each Region? e Types of factors (internal and external) that have 
e Are there unintended (positive or negative) supported or hindered progress towards outcomes 
effects emerging? overall and in each region. 


e What (common and context specific) factors 
support or hinder progress? Why? 


5.3 CLEAR e ls there early evidence that CLEAR’s e Evidence of Regional Centres having contributed to 
contributions to activities and outcomes are likely to developing /strengthening context-specific M&E 
higher/highest contribute to strengthened M&E systems 
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Part A - Summary of CLEAR Progress at Mid-Term 


level Outcomes 


5.4 Conditions for 
sustainability of 
CLEAR 
programming at 
the regional level 


5.5 Sustainability 


and practices in the targeted regions? 


Is there early evidence that CLEAR’s 
activities and outcomes are likely to 
contribute to stakeholders’ use of evidence 
in making decisions for improved 
development results? 


How do different stakeholders define 
sustainability and to what extent are their 
goals in this regard realistic? 


How likely are regional programme activities 
and results to be sustained and dynamically 
adapted? Why? 

What actions have the Centres taken to 
sustain the activities undertaken at the 
regional level? 

What types of (explicit and implicit) signals 
and incentives aimed at fostering 
sustainability have been set out by different 
stakeholders? With what effect? 


How likely is the work being done by 
CLEAR Centres to be sustained within their 
host institutions? Why? 


How likely is the CLEAR network to be 


systems. 


Evidence of consulted CLEAR clients using the 
knowledge, skills, or information they gained 
through CLEAR activities to improve evaluation 
practice in their respective organisation, country, or 
region. 


Evidence of centre strategic clients and 
stakeholders reporting increased use of evidence in 
decision making for development results. 


CLEAR RC staff, clients, and stakeholder views on 
likelinood of regional programme activities and 
results being sustained and dynamically adapted. 


Extent to which regional/national actors express 
ownership of and demonstrate leadership for the 
objectives of the respective RC. 


Evidence of strategies, plans and resources 
allocated to sustainability planning, actions and 
monitoring within the CLEAR network and by the 
host institutions 

Stakeholder views on the effects of existing signals 
and incentives for sustainability set out by different 
stakeholders. 

% of Centre costs that are subsidized by CLEAR 
programme 

% of Centre costs that are subsidized by the Host 
institution 

Evidence that CLEAR clients are willing to pay for 
services received 


Extent to which the CLEAR funding mechanism 
(grants)and donor funding behaviours support 
medium term sustainability 

Other factors likely to support or limit the likelihood 
of regional learning outcomes being sustained. 


Evidence of strategies, plans and resources 
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Part A - Summary of CLEAR Progress at Mid-Term 


of the CLEAR 
network 


6.0. Efficiency 6.1 Scheduling 


6.2 CLEAR 
systems 


6.3 Cost of 
CLEAR services 


re 
Recommendations 
(CLEAR 
programme) 


sustained and dynamically adapted over 
time? 

What actions have the CLEAR Secretariat 
and Board taken to sustain the activities 
undertaken at the global level? 


To what extent are global and regional 
programming activities on track time wise? 


What internal and external factors have 
affected (positively or negatively) the 
scheduling? 


To what extent does CLEAR have effective 
and efficient systems in place to foster, 
encourage learning, sharing among the 
Centres and within the network? 


What additional factors have positively or 
negatively affected the efficiency of each 
Regional Centre, and of CLEAR overall 


How do the costs of CLEAR services 
compare to the costs of other service 
providers? 


What mid-term corrective actions might be 
helpful to improve the effectiveness, 
relevance, sustainability and efficiency of 
the CLEAR programme at global and 
regional levels? 


Recommendations to the CLEAR Board 


allocated to sustainability planning, actions and 
monitoring within the CLEAR network 


Evidence that members in the CLEAR network 
value and are willing to pay for or otherwise 
contribute to maintaining the benefits received 
through the CLEAR network in the medium terms 


Current characteristics of the CLEAR global network 
(including on power relations between network 
members) and implications for network 
sustainability. 


Other factors likely to support or threaten the 
sustainability of the CLEAR network. 


Comparison of planned and actual timelines for 
programme activities. 


Stakeholder views on factors (internal and external) 
that have affected scheduling. 


Perceptions of interviewed CLEAR stakeholders 


Evidence of systems in place (including, for 
example, for outsourcing M&E capacity building 
activities to other implementing partners outside the 
CLEAR network). 


Stakeholder views on key factors influencing 
CLEAR efficiency at global and regional levels. 


Comparison of CLEAR products and services with 
those similar in content, duration, and delivery 
modality provided by other suppliers. 


Stakeholder views on value for money of CLEAR 
services and products. 
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Part A - Summary of CLEAR Progress at Mid-Term 


8.1. Roles & 
Responsibilities 


8. Organisational 
Arrangements 


8.2 Governance: 
CLEAR Board 


8.3 Governance: 
Regional Centres 
and Advisory 
bodies 


Recommendations to the CLEAR 
Secretariat 


Recommendations to CLEAR Regional 
Centres 


Are roles and responsibilities of key units 
within the CLEAR network (including the 
Board, Secretariat, Centres, sub-centres, 
host institutions, Advisory bodies) clear and 
appropriate? 

Are reporting relationships and 
accountabilities among and between the 
various units in the CLEAR network clear 
and appropriate? 


How effective is the CLEAR Board in 
carrying out its strategic and administrative 
responsibilities? 

To what extent does the CLEAR Board 
adequately represent the interests of 
CLEAR stakeholders? 


How effective are CLEAR Advisory bodies in 
carrying out their responsibilities? 

To what extent do CLEAR Advisory bodies 
have the human and other resources to 
effectively to carry out Committee 
responsibilities? 

To what extent do CLEAR Advisory bodies 
adequately represent the interests of 
CLEAR stakeholders in the Regions? 


What if any, positive or negative effects 
does the respective host institution have on 


Evidence of clearly articulated individual and 
collective roles and responsibilities, accountabilities 
and reporting relationships known by all key CLEAR 
units 

Perceptions of Board, Secretariat, Centres, sub- 
centres, host institutions, Advisory bodies 


Perceptions of CLEAR Board representatives 


Perceptions of other CLEAR units (CLEAR Centres, 
Host institutions, Secretariat) 


Evidence that the Board is : 
Providing strategic direction to the Programme 
Reviewing Programme progress and effectiveness 


Fulfilling its responsibilities as regards 
communications and outreach (including 
fundraising) 

Percentage of CLEAR change agent categories that 
are represented on the Board 


Descriptions of criteria and processes used to 
identify Centres 

Perceptions of CLEAR Advisory Body 
representatives 

Perceptions of other CLEAR units (CLEAR Board, 
Centres, Host institutions, Secretariat) 


Number and composition of existing Advisory 
bodies 
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Part A - Summary of CLEAR Progress at Mid-Term 


the governance of each Centre? 


e Percentage of CLEAR change agent categories that 
are represented on the Advisory Committees 
Evidence that existing Advisory bodies are: 
Providing guidance to the Centres on their work 
programme 

e Providing guidance to the Centres on 
implementation issues 

e Number and nature of gaps in Advisory Board 
resources 

e Types of influences on Centre governance 
attributed by stakeholders to the respective host 
institution environment 


8.4 Governance: e How effective is the Secretariat in carrying e Perceptions of CLEAR Secretariat staff members 
Secretariat out its responsibilities? e Perceptions of other CLEAR units (CLEAR Board, 
e To what extent does the Secretariat have Centres, Host institutions, Advisory bodies) 


the human and other resources to effectively Evidence that the Secretariat is/nas been: 


i ibilities? 
carry out its Kapara UE Administering the Programme, based on the 
e What are benefits and drawbacks of the Board’s decisions 


Secretariat being housed in the IEG? 


Managing the selection of the Centres 


Organizing and delivering the global component of 
the Programme 


Coordinating communications and outreach 
Number and nature of Secretariat resources 


8.5 Coordination, e What aspects of the coordination, e Communication, coordination problem-solving 
collaboration, collaboration, problems-solving and mechanisms outlined and reported to be effective 
problems solving decision-making of the CLEAR network and e Examples of reported tensions in coordination, 
and decision among CLEAR units are working? Not collaboration, problem solving and/or decision 
making working? making 
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e To what extent were the criteria used to 
identify and select host institutions for 
CLEAR Centres and sub Centres 
transparent and appropriate? 


e To what extent does CLEAR have effective 
systems in place to coordinate with other 
key initiatives, institutions involved in M&E 
Capacity development? 


8.6 Monitoring and + Does the CLEAR Programme have useful e Data collected on programme delivery and 
reporting ways/standards in place to define, monitor programme administration to assess progress 
and report on the quantity and quality of against forecast 
CLEAR services and outcomes? e Evidence of use of monitoring data to take 
e Does the programme have the right corrective actions 
accountability and learning structures in e Quality and frequency of reporting on monitoring 
place to follow progress and to make data to key stakeholders 


i i i ? 
comectve acvonsde ROUES e Stakeholders’ satisfaction with reporting and 


obtaining feedback from and reporting to 
national stakeholders’ on CLEAR 


performance? 
8.7 Financial e To what extent are the receipt and flow of 
management financial resources from donors to CLEAR 


Centres and sub-centres supporting the 
realization of CLEAR objectives in timely 


ways? 
9. e What mid-term corrective actions might be 
Recommendations helpful to improve the effectiveness, 
(organisational relevance, sustainability and efficiency of 
arrangements) the CLEAR organisational arrangements? 


e Recommendations to the CLEAR Board 


e Recommendations to the CLEAR 
Secretariat 


e Recommendations to CLEAR Regional 
Centres 
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Annex D Evaluation Matrix (data collection) 


Lines of Document Review Stakeholder consultations‘? 
inquiry/ 


Methods 


Key areas 


Network Analysis 


Regional 
Centre 
specific?! 
Board 
members 
RAC22 
CLEAR 
Secretariat 
institutions 
RC Clients? 
providers & 
net-works 
Thought 
leaders on 
MEAD) 


= 
© 
= 
D 
_ 
Ko] 
D 
= 
oð 
D 
_ 
2) 
= 
v 
_ 
© 
h= 
zi 


CLEAR global/ 
overall2° 
collaborators?* 
Potential RC 


Part A: Summary of CLEAR Progress at Mid-Term 
1.1 y y y 


Establishment 
of CLEAR 
Secretariat 
and Board 


1.2 Global y y y y 
level activities 
and outputs 


2.1 Activities y y y y y y 
leading to the 

identification 

of Host 





19 In person; as well as via telephone/Skype/videoconference and via email. 

20 E.g.: CLEAR strategy, workplans, reports, meeting/event minutes, knowledge products etc. 
21 E.g.: strategies, workplans, reports, financial statements, knowledge products/studies, etc. 
22 Regional Advisory Committees. 


23 |.e. individuals and organizations/institutions that have benefited from/using CLEAR RC services and products such as training, technical assistance 
etc. Some individuals may be both clients and collaborators at the same time. In these cases the evaluation team will try to elicit information in relation 
to both perspectives. 


24 Individuals/organizations working collaboratively with RCs e.g. to conduct research or prepare/deliver M&E training or services. See previous footnote 
about possible dual lens of client and collaborator at the same time. 
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Lines of Document Review Stakeholder consultations 1° 


inquiry/ 
Methods 





Institutions 
and the 
establishment 
of Regional 
Centres 


2.2 Regional y y y y y y y ? ? 
Centre 

activities/ 

outputs 


Part B: Analysis of CLEAR Programme Level Performance 


3.1 Global y y y y y y y y 


Programming 


3.2 Regional y y y y y y y y y y y 


Programming 


4.1 Context \ y y y y y y \ \ y 


and Rationale 


4.2 Principles y y y y y y y y y y 
of inclusion 
and equality 


4.3 Targeted y y y y y y y y y y 
regions/ 

countries/ 

change 

agents 


4.4 Donors y y y 


4.5 Host y y y 
institutions 


4.6 y y y y y y y V 
Complementa 

rities with 

existing M&E 

Capacity 
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Lines of 


inquiry/ 
Methods 





Development 
initiatives 
5.1 Global 


Learning 
Outcomes 


5.2 Regional 
Learning 
Outcomes 


5.3 CLEAR 
contributions 
to 
higher/highest 
level 
Outcomes 


5.4 Conditions 
for 
sustainability 
of CLEAR 
programming 
at the regional 
level 


55 
Sustainability 
of the CLEAR 
network 


6.1 CLEAR 
systems 


6.2 Cost of 
CLEAR 
services 





8.1. Roles & 


Document Review 


y 
y 
y y 
y 
y 
y 
y y 


y 


Stakeholder consultations?? 


y 
y y 
y 
y y 
y 
y 
y 
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Lines of Document Review Stakeholder consultations '9 


inquiry/ 
Methods 
Responsibilitie 
Ss 


8.2 y V y y 


Governance: 
CLEAR Board 


8.3 y y y y y y y 
Governance: 

Regional 

Centres and 

Advisory 

bodies 


8.4 V V N y 
Governance: 
Secretariat 


8.5 y N N N y 
Coordination, 

collaboration, 

problems 

solving and 

decision 

making 


8.6 Monitoring y y y y y y y y 


and reporting 


8.7 Financial \ y y \ \ 
management 
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Annex E Draft Alternative Theory of Change 


The evaluation team developed a draft alternative Theory of Change. The ToC is deliberately 
formulated at a generic level and does not constitute an attempt to replace the existing overall 
CLEAR ToC. Instead, its main intent is to illustrate the types of assumptions that are not yet 
captured by the existing CLEAR ToC, and that CLEAR may want to explore in more depth through 
each of the regional centres. Both definitions of what ‘success’ will look like as well as related 
pathways of change are likely to differ between regions, this generic ToC . The CLEAR centres 
could therefore use this generic ToC as a starting point to develop — as suggested in the 
recommendations — their own, region-specific ToCs. 


A Revised CLEAR Theory of Change 
Figure 1 shows an overall CLEAR theory of change (ToC). It covers all the elements in the current 


CLEAR ToC, but simplified to show key rationale assumptions. 
The CLEAR Capacity Building (CB) outputs have been grouped into three groups: 


Professional development 
e Training 
e Knowledge exchange (workshops, etc.) 
e Leadership development 
e Knowledge resources 


e Networks/COP/partnership/association development 


Advisory/Consulting 
e Diagnostics 
e Evaluations, assessments, project advisory 


e Technical assistance (TA) and organization capacity building 


Promotion 
e Advocacy, promotion 
e Grants, competitions and awards 


The actors involved (stakeholders) are parliaments, government organizations, NGOs and CSOs. 
In each case, decisions are made by managers in the organizations, as well as by staff delivering 
programs and projects. In all cases, the overall CLEAR aim is that these decision-makers use 
M&E-based evidence to inform their decision making. 


The rationale assumptions for CLEAR are that 


e IF CB is provided, THEN M&E evidence can be provided, AND demand for M&E evidence 
for decision-making can be enhanced. 


e IF M&E evidence is available to decision makers, THEN they will use it to inform decisions. 


Some reading of the CLEAR material suggests that the rationale assumption is that IF the supply 
of M&E is enhanced, THEN it will be used for decision making. But this assumption is problematic, 
as considerable evidence makes it clear that just supplying information in no way ensures that it 





e-Pact 34 


CLEAR Midterm Evaluation — Annexes 





will get used. More detailed reading of the CLEAR material brings forth the more reasonable, albeit 
still quite challenging rationale assumptions above. 


Figure 1 Overall CLEAR Theory of Change 


Managers and staff use evidence in making 
decisions for improved development results 


Supply Demand 











Quality and timely M&E Strengthened 
evidence produced: M&E systems and 
practices 








e Strengthened capacity to 
produce 


CLEAR Supply and Demand Theories of Change 


Figures 2 and 3 below illustrate and discuss two nested ToC for CLEAR: one focused on the 
supply of M&E evidence and the other on building demand and an enabling environment for M&E. 
The context in all cases is an organization (government that has been receiving CLEAR capacity 
building (CB) products. 


The causal link assumptions in these ToC are the necessary events and conditions that need to 
occur for the link to be (likely) realized. 
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Figure 2 CLEAR Supply Theory of Change 


M&E 
innovations 
produced 


Assumptions (A3b) 


e Local and regional 
context provides 


Discussion 





Demand 
ToC 


Managers and staff 


receive credible and 
timely evidence 


Assumptions (A4) 





e M&E systems in 
place 


Quality and timely 
M&E evidence 
produced 


e Time and resources 
available for M&E practices 


A emoun oi Ca hes 


Staff and M&E specialists 
knowledge, attitudes, skills, 
opportunities enhanced 


T eat i 


Assumptions (A2) 


e Staff willing to learn and 
consider changing ways of 


Reach: working 


Staff and M&E 
specialists 


D a Assumptions (A1) 


M&E capacity building outputs for 
staff and M&E specialists 


e Staff and M&E specialists ] 
receive the CB products i 


e Professional development events 


The rationale assumption here is that CB efforts to enhance the production of M&E evidence will 
results in quality M&E evidence being available to managers and staff in the organization. This is a 
reasonable assumption. Supporting evidence would be that there is extensive experience and 
understanding on M& practices available to develop the CB messages and material. 


Consider the links: 


1.From outputs to reach. 
Causal assumption A1 is that the CB outputs are delivered to staff and M&E specialists. Over time 
one would hope that a significant number of staff receive the M&E training and that there are 


follow-up CB events. 
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2. From reach to enhanced capacity on M&E 
A2 assumes that there is indeed willingness to learn on the part of the staff. 


3. From enhanced capacity to useful M&E evidence 

This link is more challenging: have the CB activities resulted in more and better M&E practice? For 
that to happen (A3a) resources for producing M&E need to be forthcoming, there has to have been 
an adequate amount of CB activities and planning for the production of M&E needs to be robust 
enough, so that quality and timely M&E is delivered. Timely in the sense that M&E evidence is 
available when decisions are to be made. Evidence suggests this is a challenging assumption. 


For M&E innovations to emerge, there is a need to emphasis the local context (A3b). 

4. From M&E evidence available to it reaching decision makers 

This is another non-trivial link. It requires (A4) that there are systems and procedures in the 
organization to ensure the M&E evidence, in a digestible form, reaches managers and staff on 
time. The link is also part of the demand ToC. 

In reviewing CLEAR, one could check: 


A1: The numbers and future plans for CB in each organization involved. An issue may be: Are 
there CB efforts for M&E specialists in the organization? 


A2: The extent to which the local context is taken into account in the CB activities and there is 
emphasis on the usefulness of M&E for managing/delivering. 


A3a: Has the CB activities emphasised the importance of getting the timing right? Is there a robust 
M&E planning system in place (or being out in place) in the organization? 


A3b: Has context been adequately focused on? 


A4: Are there CB activities addressing the need for supporting systems and procedures? 
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Figure 3 CLEAR Demand Theory of Change 


Managers use Donne rs 
evidence in Assumptions (A5) 


decision making 


how decisions are made 


e Managers willing to change 


Assumptions (A4a) 


Managers receive i 
quality and timely | 
evidence i e Credible, relevant, 


A ER ER EEEE EE EERTSE 


Assumptions (A4b) 





understandable and 


e M&E systems are working timely evidence is 









available 


Risks (R1) 


Enhanced enabling environment | 
such as stronger: = = | 
A l e 
Assumptions (A3a) 


e M&E requirements 






e Managers willing to change 


Assumptions (A3b) their management practices 





Managers’ knowledge, 
e The managers with the | attitudes, skills, E IS CENTS ae En 
enhanced CB capacity | 


can bring about 


opportunities enhanced Assumptions (A2) 


e Managers willing to learn 
and consider changing ways 
of working 


Reach: 


Management | = = £4 


Assumptions (A1) 


e Key senior and other 
managers receive the CB 
products 


M&E capacity building outputs for 
managers 


e Professional development events e CB messages/approaches 


seen worth considering 


e Advocacy/promotional 'campaigns' 





Discussion 


The rationale assumption for this component of CLEAR is that 


e IF senior and other managers in an organization are informed about the benefits and 
practicalities of Má evidence, 


e THEN they will work to ensure that evidence is available for consideration when decisions 
are being made. 





e-Pact 38 


CLEAR Midterm Evaluation — Annexes 





This is, of course, a rather significant assumption for which there is not a lot of evidence. As 
discussed below, there are a host of reasons why this rationale assumption may not hold, or hold 
quite weakly. 


Consider the steps in the above ToC: 


1. From delivering capacity building (CB) outputs to reach the right officials in an 
organization. 


To get evidence more used in an organization requires that managers who are making decisions 
understand M&E and what it takes for an organization to use M&E. these understandings are what 
is expected from the CB activities, such as leadership development, training in the concepts of 
M&E, and hearing advice. 


The first step in this ToC is to get to the ‘right’ managers in the organization. Mid-level managers 
may be able to make some use of evidence in their own domain (although that could be a 
problem), but it will take senior managers (and key senior managers) to make changes in systems 
and procedures in the organization and visibly support the use of evidence in the organization. The 
causal assumptions (A1) here are that those key managers are indeed reached by the CB efforts 
and that what they first ‘see’ seems reasonable and worth considering. 


2. From reach to changing managers understanding of and their role in M&E 


Step 2 speaks to the effectiveness of the CB outputs. Managers may be attending CB events 
because they were told to do so. A significant part of the messaging would need to convince 
managers of the practical benefits of enhancing M&E, turning skeptics into those at least willing to 
learn and perhaps change their practices. Causal link assumption A2 noted the need for this 
willingness, as well as the need for the CB messages to be seen as doable in their organization. 


3. From enhanced capacity to managers calling for evidence and changing the enabling 
environment 


Understanding how they might use M&E evidence does not automatically transfer into managers 
changing their behaviour. They need to start managing on more than experience and gut feel, and 
(A3a) ask that they are provided with relevant M&E evidence to inform upcoming decisions. This is 
a critical assumption; managers may not feel comfortable in asking for such information or 
concerned that it will not support positions they have or want to hold. This is where leadership in 
the organization needs to set an example. 


It is also usually required in an organization that the enabling environment for the production, 
demand and use of M&E evidence be strengthened. This might include requirements for 
undertaking M&E and for reporting on results achieved. A key causal link assumption (A3b) is that 
those who receive the CB outputs can themselves or by persuading others bring about the needed 
procedural changes in the organization. 


4. From calling for M&E evidence to receiving it in a useable form 


Getting managers to call for M&E evidence is important, ensuring they receive quality (relevant, 
understandable, credible) and timely information is not straightforward and where the CLEAR 
supply efforts (A4a) come in. 


Equally important is that the M&E evidence reaches decision makers in a timely fashion. There us 
a history of M&E efforts missing decision windows, coming in too late to be realistically considered. 
M&E systems (A4b) for planning need to be well interested with decision processes. 
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The discussion has not focused on risks to the causal links occurring, other than the risk that an 
assumption does not occur. Here one risk is mentioned, namely (R1) that if the organization 
undergoes frequent changes in directions and/or priorities, M&E evidence on ‘past’ activities may 
not be able to contribute much to decision making, undermining the credibility of the M&E efforts.. 


5. From receiving M&E evidence to actually using it 


If all the other steps in the causal chain have been met, then is seems likely that managers will 
indeed use the M&E evidence to inform their decision making. The term inform has been stressed 
her less the image is that with M&E evidence decisions become mechanical. That is rarely the 
case. A5 requires changed decision making behaviour. The issue was mentioned earlier: 
managers may not feel comfortable in asking for such information or concerned that it will not 
support positions they have or want to hold. 
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AnnexF List of Consulted Stakeholders 


F.1 Global 


BICHSEL, Anne Senior Evaluation Swiss Agency for Donor/dev. CLEAR 
Advisor Development and Organisation donor 
Cooperation 
BOEHMER, Retired, Former Independent Evaluation Donor/dev. CLEAR 
Hans-Martin CLEAR Board Chair Group (IEG), World Bank Organisation donor 
and former Senior 
Manager of the 
Independent 
Evaluation Group 
BOWMAN, Director, Office of Development Donor/dev. CLEAR 
Debbie Evaluations and Effectiveness, Department organization donor 
Communications of Foreign Affairs and 
Trade, Australia 
COURTNEY, Sector Manager, World Bank Donor/dev. Other 
Bruce Africa Region organisation 
GERRARD, Chris Retired, Former World Bank Donor/dev. Other 
Director of Global organisation 
Partnership 
Programs 
GRAY, Cheryl Director, Office of Inter-American Donor/dev. CLEAR 
Evaluation and Development Bank organisation donor 
Oversight 
HEIDER, Caroline Director General, Independent Evaluation Donor/dev. Other 
Evaluation and Sr. Group, World Bank organisation 
VP 
HELTBERG, Lead Evaluation World Bank Donor/dev. Other 
Rasmus Officer - Global organisation 
Programs 
HOPWOOD, lan Retired Former UNICEF CSO/NGO Thought 
Representative in Senegal leader 
and Member of the 
Senegalese Evaluation 
Association (SENEVAL) 
JENSEN, Anders Senior Evaluation World Bank Donor/dev. Other 
Officer, Africa organisation 
Region 
MACPHERSON, Managing Director, Rockefeller Foundation Donor/dev. CLEAR 
Nancy Evaluation organisation donor 
MANAI, Division Manager, African Development Bank Donor/dev. CLEAR 
Mohamed Hedi Project and (AfDB) organisation donor 


Program, Evaluation 
Division, Operations 
Evaluation 

Department (OPEV) 
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NANGIA, Rakesh 


OFIR, Zenda 


PECK, Lennart 


PLUMMER, Kellie 


RIDER-SMITH, 
David 


RIST, Ray 


ROBIN, Elizabeth 


RUGH, Jim 


SCOBENA 
Richard 


WHITE, Howard 


YORK, Nicholas 
David 


Director of the 
Operations 
Evaluation 
Department, ADB 


Independent 
Consultant and 
International 
Evaluator 


Evaluation Manager 


Evaluation Manager 


Acting Head of 
Capacity 8 Quality 
Group , Evaluation 
Department, 
Evaluation Adviser 


Co-Director of 
International 
Program for 
Development 
Evaluation Training 
(IPDET) and former 
President of IDEAS 


Retired, Former 
Evaluation Capacity 
Advisor 


Coordinator 


Senior Advisor and 
Deputy Director 


Executive Director 


Director, Country, 
Corporate and 
Global Evaluation 


F.2 Anglophone Africa 


ADAMS, Samuel 


AMOATEY TEYE, 


Lecturer 


Lecturer/ CLEAR 
Programme 


African Development Bank 


(AfDB) 


Past President of the African 


Evaluation Association 


(AfrEA), former board member 
of the American Evaluation 
Association and the NONIE 


Steering Committee 
SIDA 


Office of Development 
Effectiveness, Department 
of Foreign Affairs and 


Trade, Australia 


Department for 


International Development 


(DFID) 


International Program for 
Development Evaluation 


Training (IPDET) 


Department for 


International Development 


(DFID) 


EvalPartners International 


Initiative 


Independent Evaluation 
Group (IEG), World Bank 


International Initiative for 
Impact Evaluation (SIE) 


Independent Evaluation 
Group (IEG), World Bank 


Ghana Institute of Public 


Administration (Sub- 
centre) 


Ghana Institute of Public 


Administration (Sub- 


Donor/dev. 
organisation 


CSO/NGO 


Donor/dev. 
organisation 


Donor/dev. 
organisation 


Donor/dev. 
organisation 


Donor/dev. 
organisation 


Donor/dev. 
organisation 


Donor/dev. 
organisation 


Donor/dev. 
organisation 


Donor/dev. 
organisation 


Donor/dev. 
organisation 


Academic/ 
Research 


Academic/ 
Research 


CLEAR 
donor 


Thought 
leader 


CLEAR 
donor 


CLEAR 
donor 


CLEAR 
donor 


Thought 
leader 


CLEAR 
donor 


Thought 
leader 


Other 
Thought 


leader 
Other 


Actual 
partner 


Actual 
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Charles Coordinator forthe centre) partner 
Anglophone West 
Africa Region 
BAGAKA, Obuya Senior Principal Kenya School of Academic/ Actual 
Lecturer/ CLEAR Government Research partner 
Africa East African (Sub-Centre) 
Regional 
Coordinator 
BASSON, Former Chair South African Monitoring NGO/CSO Actual 
Raymond and Evaluation Association partner 
(SAMEA) 
BISGRAD, Director Khulisa Management Other CLEAR 
Jennifer Services (Pty) Ltd- competitor 
Collaborator 
BLET, Cyril Consultant African Development Bank Donor/Devel Actual 
opment partner 
organisation 
BONSU, Samuel Dean Ghana Institute of Public Academic/ Actual 
K. Administration Business Research partner 
School, Ghana 
BYAMUGISHA, Commissioner, Office of the Prime Minister | Academic/ Actual client 
Albert Monitoring & of Uganda Research 
Evaluation 
CLOETE, Fanie Professor : Dept. of University of Johannesburg Academic/ CLEAR 
Public Management Research competitor 
and Governance 
CLYNICK, Tim Interim Director Clear Anglophone Africa Academic/ CLEAR 
Centre Research faculty & staff 
CRAWLEY, Researcher CLEAR Anglophone Africa  Academic/ CLEAR 
Kieron Centre Research faculty & staff 
EVERETT, Mark Employee of DPME Dept. of Performance, Government Actual 
and alumnus at Monitoring and Evaluation (national) partner & 
WITS — The Presidency, Republic client 
of South Africa 
FITZGERALD, Special Advisor CLEAR Anglophone Africa  Academic/ CLEAR 
Patrick Centre Research faculty & staff 
GACHIRE, Betty Senior Lecturer Kenya School of Academic/ Actual 
Muthoni Government Research partner 
GOLDMAN, lan Head of Evaluation Dept. of Performance, Government Actual 
and Research Monitoring and Evaluation (national) partner & 
— The Presidency, Republic client 
of South Africa 
GOVENDER, Finance Office CLEAR Anglophone Africa  Academic/ CLEAR 
Vigie Centre Research faculty & staff 
GRIESSEL, Head of Evaluation Gauteng Government Government Actual client 
Annette (sub- 
national) 
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HEUSTICE, Director Info4 Africa HIV Networking NGO/CSO Actual client 
Debbie Organisation 
INAMBAO, Staff CLEAR- Anglophone Africa Academic/ CLEAR 
Amusaa Centre Research faculty & staff 
JACKSON, President, Senior E.T. Jackson & Associates Other Actual 
Edward T. management partner 
consultant 
JACOBS, Christel Evaluation and Dept. of Performance, Government Actual 
Research Unit : Monitoring and Evaluation (national) partner 8 
Capacity — The Presidency, Republic client 
Development of South Africa 
KIBOI, David Chief Economist Office of the Prime Minister: Government Actual Client 
Ministry of State for (national) 
Planning, National 
Development and Vision 
2030, Kenya 
KITSHOFF, Ruan Governance German Cooperation (GIZ)  Donor/ Actual 
Support Development partner 
Programme organisation 
LATIB, Salim Staff CLEAR- Anglophone Africa  Academic/ CLEAR 
Centre Research faculty & staff 
LESLIE, Michael Consultant PDG (private consulting Other CLEAR 
firm, South Africa) competitor 
MACHUKA, Director of Office of the Prime Minister: Government Actual 
Samson Planning, Ministry of State for (national) partner & 
Monitoring and Planning, National client 
Evaluation Development and Vision 
2030 Kenya 
MAINA, Rachel Principal Lecturer Kenya School of Academic/ Actual 
Ngesa Government Research partner 
MALUKA, Employee of DPME Dept. of Performance, Government Actual 
Bongani and former students Monitoring and Evaluation (national) partner & 
— The Presidency, Republic client 
of South Africa 
MANU, Franklyn Rector Ghana Institute of Public Academic/ Host 
A. Administration Research institution 
MCLENNAN, Research Director University of the Academic/ Host 
Anne Witwatersrand (WITS) Research institution 
MHLANTLA, Former CLEAR University of the Academic/ CLEAR 
Thandeka Staff Research Witwatersrand (WITS) Research faculty & staff 
assistant 
MOGALE, Head of School PADM, University of the Academic/ Host 
Thomas Witwatersrand (WITS) Research institution 
MOODLEY, Director Palmer Development Other CLEAR 
Nishendra Group (private consulting competitor 
firm, South Africa) 
MOORE, Rob Deputy Vice- University of the Academic/ Host 
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Chancellor, Witwatersrand (WITS) Research institution 
Advancement and 
Partnerships 


Division 
MWAULUKA, Staff Monitoring and Evaluation Government Potential 
Winza Department (national) client 
Economic Management 
and Finance 
Ministry of Finance 
Government of Zambia 
NTUKUMBA, Responsible for Dept. Of Performance, Government Actual 
Stanley M&E capacity Monitoring and Evaluation (national) partner & 
building — The Presidency, Republic client 
of South Africa 
NOWOTTNY, Head of Strategy Civicus CSO/NGO Potential 
Mark partner 
OFIR, Zenda International School of Public Academic/ Thought 
Evaluator & Leadership, University of Research leader 
Honorary Professor Stellenbosch 
OLIECH, Daniel Ag. Head of Kenya School of Academic/ Actual 
Research and Government Research partner 
Consultancy 
OLOO, Adams Chairman/ CLEAR Dept. of Political Science Academic/ Thought 
G.R. Regional Advisory and Public Administration, Research leader 
Committee Member University of Nairobi, Kenya 
OYORE, John Lecturer and Department of Community  Academic/ Actual 
Paul Chairman Health, School of Public Research partner 
Health, Kenyatta University, 
Kenya 
PILLAY, Pundy Professor Public University of the Academic/ Host 
Finance Witwatersrand (WITS) Research institution 
PORTER, Former Director CLEAR Anglophone Africa  Academic/ CLEAR faculty 
Stephen Centre Research & staff 
POSWELL, Laura Director JPAL South Africa Academic/ CLEAR 
CES competitor 
RABIE, Babette Former Chair South African Monitoring CSO/NGO Actual partner 
and Evaluation Association 
(SAMEA) 
RANKIN, Neil Consultant Impact University of Stellenbosch Academic/ CLEAR 
Evaluation Research competitor 
ROBERTS, Researcher, African School of Economics and Academic/ CLEAR faculty 
Gareth Micro-Economics Business Sciences , Research & staff 
Research Unit CLEAR Anglophone Africa 
(AMERU) Centre 
SCHOER, Volker  Lecturer School of Economics and Academic/ Host institution 
Business Sciences Research 
SENABE, Sipho Chief Director HIV Department of Public Government Actual client 
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Workplace Policy 


TETTEH, Gifty Strategic Outreach 


Officer 


TIMM, Jonathan Director, Citizens- 


Based Monitoring 


VAWDA, Ahmed Outcome Facilitator 


VENTER, Janet Programme Officer 


WACHIRA, Elijah Ag. Director 

K. 

WOTELA, M&E Graduate 

Kambidima Program Convenor 
and Leader 

YIRENKYI- Lecturer 

FIANKO, Adowa 

ZONDI, Thulile Technical Expert for 
CLEAR-AA and 
Senior Lecturer at 
WITS 

ZUMA, Thabi Employee of DPME 


and former student 


F.3 East Asia 


Service and Administration, (national) 

South Africa 

Episcopal Relief & CSO/NGO Actual client 
Development (NGO) , 

Ghana 

Dept. of Performance, Government Actual 
Monitoring and Evaluation (national) partner & 

— The Presidency, Republic client 

of South Africa 

Dept. of Performance, Government Actual partner 
Monitoring and Evaluation (national) & client 

— The Presidency, Republic 

of South Africa 

Japan International Donor/Devel Actual client 
Cooperation Agency (JICA opment 

South Africa Office) organisation 

Kenya School of Academic/ Actual 
Government Research partner 
University of the Academic/ Host 
Witwatersrand (WITS) Research institution 
Ghana Institute of Public Academic/ Actual 
Administration Research partner 
CLEAR Anglophone Africa  Academic/ CLEAR 
Centre Research faculty & staff 
Dept. of Performance, Government Actual partner 
Monitoring and Evaluation (national) & client 


— The Presidency, Republic 
of South Africa 





CHEN Qihui Assistant Professor; China Agriculture Academic/ CLEAR EA 
Impact Evaluation University Research Faculty 
Instructor 
CHEN Yimei Executive Director China Development Brief CSO Potential 
client 
DENG Guosheng Director and The Center for Innovation Academic Thought 
Professor and Social Responsibility, leader 
Tsinghua University of 
China 
GAO Ruirui M&E Researcher Social Resources Institute CSO CLEAR 
Competitor 
GOU Yannan Professor Fudan University of China  Academic/ Thought 
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GU Qing 


GUANG Xiuzhen 


GUO Ran 


HE Xiaojun 


HUANG Jian 


KANG Xiaoguang 


KHATTRI, Nidhi 


LI Kouaing 


LI Zhiyan 


LIU, Scott 


LUO Jie 


MORRA IMAS, 
Linda 

MSUYA, Joyce 
MU Tingting 


NG BOW, Lisa 


PENG Runzhong 


RAUNIYA, 
Ganesh 


WANG Haitao 


WEST-MEIERS, 
Maurya 


Team Leader 


Director 


M&E Researcher 


RAC member 


Manager 


Director and 
Professor 


Lead Evaluation 
Officer, Head of 
CLEAR Secretariat 


Director General, 
AFDC 


Director 


Senior Project 
Officer, AFDC 


Deputy Director 


SHIPDET Instructor 


Regional 
Coordinator 


Officer 


Chief, Planning, 
Monitoring and 
Evaluation 


Director and 
Professor, AFDC 


Principal Evaluation 
Specialist 


Director 


Evaluation Officer, 


UNDP China 


Ministry of Finance China 
IFI Division 


Social Resources Institute 


International Poverty 
Reduction Center in China 


China Construction Bank 
Shanghai Branch 


NPO Research Center, 
Renmin University of China 


IEG, World Bank 


CLEAR East Asia 
Social Resources Institute 
CLEAR East Asia 


Division of Performance 
Evaluation, Shanghai 
Municipal Finance Bureau, 
China 


IDEAS 
World Bank Institute 


Ministry of Finance China 
IFI Division 


UNICEF Office for China 


CLEAR East Asia 


Independent Evaluation 
Division of Asia 
Development Bank 


Ministry of Finance China 
Performance Budgeting 
Division 

IEG, World Bank 


Research 


Development 
org. 
Government 
CSO 
Government 
Other-private 
sector 


Academic 


CLEAR 
Secretariat 


Academic 


CSO 


Academic 


Government 
(sub 
national) 


Individual 
Donor/develo 
pment org. 


Government 


Donor/develo 
pment org. 


Academic 


Donor/develo 
pment org. 


Government 


CLEAR 
Secretariat 


leader 


Potential 
client 


Host 
Institution 


CLEAR 
Competitor 


RAC 


Client 


Thought 
leader 


CLEAR staff 


Host 
institution 


CLEAR 
Competitor 


Host 
institution 


Client 


CLEAR EA 
Faculty 


Partner 


Host 
Institution 


Potential 
Client 


CLEAR EA 
Faculty 


Partner 


Host 
institution 


CLEAR staff 
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WU, Annie 


YAO Chuanjin 


YE Hong 


ZHAO Min 


Program 
Coordinator, AFDC 


Director and 
Professor 


Deputy Director 


Associate 
Professor, AFDC 


F.4 Francophone Africa 


Senegal 


CLEAR East Asia 


Research Centre for 
Philanthropy and Social 


Enterprise; Beijing Normal 


University of China 


International Division, 
Shanghai Municipal 
Finance Bureau 


CLEAR East Asia 


Academic Host 
institution 

Academic Thought 
leader 

Government Client 

(sub 

national) 

Academic CLEAR EA 
Faculty 





AW, Boubacar Trainer & Monitoring Centre Africain d'Études Academic CLEAR 
& Evaluation Expert, Supérieures en Gestion faculty 
new coordinator of | (CESAG) 

CLEAR FA 

BAIDARI, Director General Centre Africain d’Etudes Academic Host 

Boubacar Senior Lecturer Supérieures en Gestion institution 
(CESAG) 

BAKOYE, Deputy Coordinator Secrétariat Permanent de Government Potential 

Saádou la Cellule de Suivi de (national) client 
l'Action Gouvernementale 
(Niger) 

BALDE, Séga Head of Department Centre Africain d'Études Academic Host 
Supérieures en Gestion institution 
(CESAG) 

CHABI, Bertin Head of Department Centre Africain d'Études Academic Host 
Supérieures en Gestion institution 
(CESAG) 

COULIBALY, Head of Department Centre Africain d'Études Academic Host 

Abdoulaye Supérieures en Gestion institution 
(CESAG) 

DIA, Ousmane Consultant, n/a Individual CLEAR 
Developed faculty 
communications 
plan for CLEAR 
Francophone Africa 

DIENG, Associate Director Ministry of Economy and Government Potential 

Abdoulaye Finance, Government of (national) client 

Senegal 

DIOP, Cheikh Monitoring & Ministry of Economy and Government Potential 

Evaluation Unit Finance, Government of (national) client 
Senegal 
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DIOP, El Hadj 
Mamadou 


DIOP, Moustapha 
DJIDJOHO, 


Aristide 
FALL, Awa Nguer 


FALL, lbrahima 
Mboulé 


FALL, Ngounda 
Kane 

GERVAIS, Marie 
GNANSOUNOU, 
Sikatin 


GUENE, Racine 


GUEYE, El Hadji 


CGUEVE? 


Soukeyna Dièye 


HOUINSA, David 


KABORE, 


Inoussa 


KOFFI, Amani 


KOUASSI, Alexis 


LAHEY, Robert 


LEYE, Mamadou 


Consultant 


Trainer 


Coordinator 


Trainer 


General Secretary 


External Consultant 


Head of Department 


Head of Department 


(former) Coordinator 
of CLEAR Program 
Francophone Africa 


Program Assistant 


Consultant, 
Monitoring and 
Evaluation 


Regional Chief 
Monitoring and 
Evaluation 


Head of Department 


Head of Quality 
Assurance Unit 


External Consultant 


Head of service 


Directorate General of 
Finance (DB/DGF), 
Ministry of Economy and 
Finance, Government of 
Senegal 


CLEAR Francophone 
Africa Centre 


Bureau d'Évaluation de 
l'Action Publique (Bénin) 
Caisse d’Epargne 


Populaire et 
d'investissement (CEPI) 


CLEAR Francophone 
Africa Centre 


Ministry of Economy and 
Finance, Government of 
Senegal 


Laval University 


Centre Africain d'Etudes 
Superieures en Gestion 
(CESAG) 


Centre Africain d'Etudes 
Superieures en Gestion 
(CESAG) 


CLEAR Francophone 
Africa Centre 


Centre Africain d'Etudes 
Superieures en Gestion 
(CESAG) 


n/a 


UNICEF, West & Central 
Africa Regional Office 


Centre Africain d'Etudes 
Superieures en Gestion 
(CESAG) 


Centre Africain d'Etudes 
Superieures en Gestion 
(CESAG) 


Consultant, Ottawa, 
Canada 


Centre Africain d'Etudes 
Superieures en Gestion 
(CESAG) 


Government 
(national) 


Academic 
Government 
(national) 
Other - 
private sector 


Academic 


Government 
(national) 


Academic 


Academic 


Academic 


Academic 


Academic 


Individual 


Donor/dev 


org 


Academia 


Academic 


Individual 


Academic 


Potential 
client 


CLEAR staff 


Potential 
client 


Potential 
client 


CLEAR staff 


Potential 
client 


Clear faculty 


Host 
institution 


Host 
institution 


Host 
institution 


Host 
institution 


CLEAR 
faculty 


Potential 
partner 


Host 
institution 


Host 
institution 


CLEAR 
faculty 


Host 
institution 
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LO, Abdou Karim 


LO, Aïssatou 


LO, lbrahima 
Thierno 


MENARD, Amos 


MONI, Audry 


NANA, Cathérine 


NDIAYE, Sarr 
Sylla 


NDIAYE, 
Dieynaba Dia 


NDIAYE, 
Ibrahima 


OULD DAHI, 
Abdel Aziz 


SARR, 
Madjiguène 
Khady S. 


SENE, Adama 
Guiro 


SENE, Papa 
Ndiamé 


SOMBIE, Issa 


SOW, Moctar 


TALL, Abdoul 
Aziz 


Public Management 
Consultant, Member 
of SENEVAL, 

Associate Professor 


Administrative 
Assistant 


Knowledge 
Management Office, 
Planning, 

Monitoring & 
Evaluation Section 
Assistant 


Head of service 


Head of Department 


Monitoring and 
Evaluation Officer 


Assistant-Treasurer 


Director-General 


Consultant, 
Monitoring and 
Evaluation 


Recovery Advisor 
(Finance) 


Trainer 


Director 


Trainer 


President 


Director 


Centre Africain d'Etudes 
Superieures en Gestion 
(CESAG) 


CLEAR Francophone 
Africa Centre 


UNICEF West & Central 
Africa Regional Office 


CLEAR Francophone 
Africa Centre 


Centre Africain d'Etudes 
Superieures en Gestion 
(CESAG) 


Centre Africain d'Etudes 
Superieures en Gestion 
(CESAG) 


FHI360, USAID/ 
Programme Gouvernance 
et Paix au Senegal 


Senegalese Evaluation 
Association (SenEval) 


Bureau Organisation et 
Méthodes (BOM) 
Government of Senegal 


Centre Africain d'Etudes 
Superieures en Gestion 
(CESAG) 


CLEAR Francophone 
Africa Centre 


Directorate General of 
Finance (DB/DGF), 
Ministry of Economy and 
Finance, Government of 
Senegal 


[SES RE 


Senegalese Evaluation 
Association (SenEval) 


Cabinet de la présidence 
(Senegal) 


NGO/CSO 


Academic 


Donor/Dev 
org 


Academic 


Academic 


Academic 


Government 
(national) 


NGO/CSO 


Government 
(national) 


Individual 


Academic 


Academic 


Government 
(national) 


Academic 


NGO/CSO 


Government 
(national) 


Potential 
partner 


CLEAR staff 


Potential 
partner 


CLEAR staff 


Host 
institution 


Host 
institution 


Potential 
client 


Potential 
partner 


Potential 
client 


CLEAR 
faculty 


Host 
institution 


CLEAR staff 


Potential 
client 


Potential 
partner 


Potential 
partner 


Potential 
client 
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TALL, Oumoul 
Khairy Ba 


TANO BEUGRE, 
Justine A. 


THIAM, Diarra 


THIAM, 
Mamadou 
Moustapha 


TRAORE, 
Ahmadou 


TREVINO, 
Monica 


AZENIA, Laurent 


BADOLO, 
Celestin 


DEJONGH, Guy 


DIARRA, 
Abdoulaye 


DIEUDONNE B., 
Kini 


DJIM DOUMBE, 
Damba 


GUEMADJI- 
GBEDEMAH, 
Tété Eyon 


KARAMBIRI, 
Harouna 


KOANDA, Mady 


KOBIANE, Jean- 
François 


Trainer & Evaluator 


Secretary General 


Head of Department 


Trainer 


Consultant, 
Monitoring and 
Evaluation 


External Consultant 


Officer, Partnership 
Development 
(Reporting des 
conventions et 
accords de 
financement) 


Director of 
Programs 


Monitoring & 
Evaluation 
Specialist 


Professor, 
Researcher 


Analyst, 
Governance 
Program 


Educational Officer 


Professor, 
Researcher 


Director of Doctoral 
Studies 


Director of CLEAR- 
2iE 


Director 


OKT Consult 


Centre Africain d'Etudes 
Superieures en Gestion 
(CESAG) 


Centre Africain d'Etudes 
Superieures en Gestion 
(CESAG) 


Université Cheikh Anta 
Diop (UCAD) 


n/a 


Consultante, Montreal, 
Canada 


2iE Foundation 


Centre de suivi et 
d'analyses citoyens des 
politiques publiques, 
Ouagadougou, Burkina 
Faso 


UNICEF, Ouagadougou, 
Burkina Faso 


2iE Foundation 


UNDP, Ouagadougou, 
Burkina Faso 


2iE Foundation 


2iE Foundation 


2¡E Foundation 
2¡E Foundation 


Institut Supérieur des 
Sciences de la population, 


Private 
sector 


Academic 


Academic 


Academic 


Individual 


Individual 


Academic 


NGO/CSO 


Donor/dev 
org 


Academic 


Donor/dev 
org 


Academic 


Academic 


Academic 


Academic 


Academic 


Competitor 


Host 
institution 


Host 
institution 


CLEAR 
faculty 


CLEAR 
faculty 


Other 


Potential 
partner 


Potential 
partner 


Potential 
partner 


Potential 
partner 


Potential 
partner 


Potential 
partner 


Potential 
partner 


Potential 
partner 


Potential 
partner 


Potential 
partner 
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KOUAME, 
Kouassi 


KOUANDA, Seni 


LANKOANDE, 
Gountiéni Damien 


MAIGA, Amadou 
Hama 


OUEDRAOGO, 
Michel 


OUEDRAOGO, 
Michel 


OUEDRAOGO, 
Salifou 


POUAHOUKIGA, 
Anébakouri 
Ezéchiel 


SAWADOGO, 
Martine 


SEMPORE, 
Francis 


TRAORE, 
Noumoutié 
Herbert 


ZOUNGRANA, 
Salifou 


COULIBALY, 
Mamadou 


Secretary General 


Deputy Director 


Executive Secretary 


Interim Director 
General 


Research Assistant 


Secretary General 


Director, West 
Africa 


Deputy Head of 
Science Committee 


Professor, 
Researcher 


Counsel to the 
Director General ; 
Officer, Digital 
Services and 
Forecasts (Chargé 


du Numérique et de 


la Prospective) 


First President 


Monitoring & 
Evaluation 
Specialist 


Permanent Secretary 


Ouagadougou 
2iE Foundation 


Institut Africain de Santé 
Publique, Ouagadougou, 
Burkina Faso 


Groupe de Recherche et 
d'Analyse Appliquées pour 
le Développement 
(GRAAD), Ouagadougou, 
Burkina Faso 


2iE Foundation 


Groupe de Recherche et 
d'Analyse Appliquées pour 
le Développement 
(GRAAD), Ouagadougou, 
Burkina Faso 


Réseau Burkinabais du 
Suivi et d’Evaluation, 
Ouagadougou, Burkina 
Faso 


SOS Sahel, Ouagadougou, 
Burkina Faso 


Réseau Burkinabais du 
Suivi et d’Evaluation, 
Ouagadougou, Burkina 
Faso 


2¡E Foundation 


2¡E Foundation 


Cour des Comptes, 
Ouagadougou, Burkina 
Faso 


UNDP, Ouagadougou, 
Burkina Faso 


RISE (Réseau lvoirien de 
Suivi et Evaluation — lvorien 
Network for Monitoring and 
Evaluation ) 


Academic 


Academic 


Academic 


Academic 


Academic 


NGO/CSO 


NGO/CSO 


NGO/CSO 


Academic 


Academic 


Government 
(national) 


Donor/dev 
org 


NGO/CSO 


Potential 
partner 


Potential 
partner 


Potential 
partner 


Potential 
partner 


Potential 
partner 


Potential 
partner 


Potential 
partner 


Potential 
partner 


Potential 
partner 


Potential 
partner 


Potential 
client 


Potential 


partner 


Potential 
partner 
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Hopwood, lan 


KEDOWIDE, 
Francois- 
Corneille 


Former director of 
M&E UNICEF 


Regional Counsellor 
for Evaluation 


Member of SenEval 


UNDP 


NGO/CSO 


Donor/dev 
org 


Potential 
partner 


Thought 
leader 
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F.5 Latin America 


ACEVEDO PINA, 
Jorge 


APARICIO, Javier 


ARELLANO 
GAULT, David 


AREVALO, Fidel 


BALLESCA, 
Monica 


BARRANTES, 
Alexandra 


BERBER CRUZ, 
Miguel Angel 


CASO, Agustin 


CASTANEDA, 
Vlado G. 


CASTRO, Manuel 
Fernando 


CEJUDO, 
Guillermo M. 


FALCON DE 
SANTA MARIA, 
Clotilde B. 


FARFAN, Gabriel 


Project Manager 


Director, Division for 


Policy Studies 


Director of the 
Division of Public 
Administration 


N/A 


M&E Director 


Social Protection 
Specialist 


Project Manager 


Former Director 
General of M&E 
Unit 


Former Director of 
the M&E Office 


Senior Evaluation 
Officer 


Provost 


Chief of the M&E 
Unit of Policies and 
Programmes 


Former Adjunct 
General Director at 
SHCP and General 
Coordinator of the 


Mexican Community 


of Practice on 
Managing for 
Development 
Results 


CLEAR Latin America 
Centre 


Centro de Investigacion y 
Docencia Económicas 
(CIDE), Mexico 


Centro de Investigación y 
Docencia Económicas 
(CIDE). Mexico 


Instituto de Investigación e 
Incidencia Ciudadana - 
INCIDENCIA-/ Secretaría 
de Planificación y 
Programación de la 
Presidencia (SEGEPLAN). 
Guatemala 


State Government of 
Jalisco, Mexico 


Organización de los 
Estados Americanos 


CLEAR Latin America 
Centre 


Secretaria de Hacienda y 
Crédito Público (SHCP) 


Ministerio de la Mujer y 
Desarrollo Social 
(MIMDES). Peru 


World Bank 


Centro de Investigación y 
Docencia Económicas 
(CIDE), Mexico 


Secretaría Técnica de la 
Presidencia. El Salvador 


Comunidad de Expertos y 
Profesionales en 
Latinoamérica y el Caribe 
en Gestión para 
Resultados en el 
Desarrollo (COPLAC- 
GprRD) 


Academic/ 
Research 


Academic/ 
Research 


Academic/ 
Research 


Government 
(national) 


Government 
(sub- 
national) 


Donor/Devel 
opment 
organisation 


Academic/Re 
search 


Government 
(national) 


Government 
(national) 


Donor/ 
Development 
organisation 


Academic/ 
Research 


Government 
(national) 


Academic/ 
Research 


CLEAR staff 
& faculty 


Actual 
partners 


Host 
institution 


Potential 
clients 


Actual clients 


Actual clients 


CLEAR staff 
8 faculty 


Actual clients 


Actual clients 


Thought 
leader 


Host 
institution 


Actual clients 


Actual 
partners 
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FAUNDEZ, Director Inclusiôn y Equidad Other Actual 
Alejandra partners 
FERNANDEZ Evaluation Officer World Bank Donor/ CLEAR staff 
ORDONEZ, and Coordinator for Development & faculty 
Ximena CLEAR-LA organisation 
FRANCO Representative in Centro Latinoamericano Other Other 
PARRILLAT, Mexico para el Desarrollo Rural 
Gerardo (RIMISP), Mexico 
GALINDEZ, Executive CLEAR Latin America Academic/ CLEAR staff 
Cristina Coordinator Centre Research & faculty 
GARCIA LOPEZ, Executive Secretary Comunidad de Expertos y Other Actual 
Roberto Profesionales en partners 
Latinoamérica y el Caribe 
en Gestiôn para 
Resultados en el 
Desarrollo/ Latin-America 
and the Caribbean 
Community of Practice on 
Managing for Development 
Results (COPLAC-GprRD) 
GONZALEZ, Monitoring and UN-Women Regional Donor/Devel Actual 
Laura Reporting Specialist Office for Latin America opment partners 
and the Caribbean organisation 
GUTIERREZ, Director Instituto Nacional de Salud Academic/Re Actual 
Juan Manuel Pública. Mexico search partners 
Centro de Investigación en 
Evaluación y Encuestas 
HEREDIA Researcher Centro de Investigación y Academic/Re Actual 
ZUBIETA, Carlos Professor, Division Docencia Económicas search partners 
of International (CIDE). Mexico 
Studies 
HERNANDEZ Secretary National Council for the Government Actual 
LICONA, Gonzalo Evaluation of Social (national) clients/partne 
Development Policy rs 
(CONEVAL). Mexico 
IRARRAZAVAL, Director Centro de Politicas Academic/Re Actual 
Ignacio Públicas UC/ Pontificia search clients/partne 
Universidad Catélica de rs 
Chile. Chile 
JAIME, Edna Director General México Evalüa 
LOPEZ Project Manager CLEAR Latin America Academic/Re CLEAR staff 
VILLAMAR, Alicia Centre search & faculty 
del Carmen 
MALDONADO, Coordinator General CLEAR Latin America Academic/Re CLEAR staff 
Claudia Centre search & faculty 
MARTINEZ Editor CLEAR Latin America Academic/Re CLEAR staff 
FIERROS, Centre search & faculty 
Alejandro 
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MARTINEZ, 
Ursula 


MESINAS 
MONTERO, 
Jorge 


MONTERO, 
Gregorio 


MONTIEL, Maria 
José 


OCEJO ROJO, 
Almudena 


OSPINA, Sonia 


PEREZ 
ECHEVERRIA, 
Manoel Alberto 


PEREZ . 
YARAHUAN, 
Gabriela 


RIOS CAZARES, 
Alejandra 


RODRIGUEZ- 
BILELLA, Pablo 


ROJAS, 
Fernando 


SALGADO, Juan 
SCO John 
SPINUZZA, 


Fernanda 


TORRES ROJO, 
Juan Manuel 


M&E Director 


Director de Calidad 
del Gasto 


General Secretary 


Project Manager 


Executive Director 


CLEAR-LA 
Regional Advisory 
Committee Member 


Project Manager 


Visiting Professor 


Researcher 
Professor, Division 
of Public 
Administration 


Executive Board 
Member 


Lead Public Sector 
Management 
Specialist for Latin 
America 


Associate 
Professor, Division 
of Legal Studies 


Researcher 
Professor, Division 
of Economics 


N/A 


Secretary General 


Ministerio de Desarrollo e 
Inclusión Social (MIDIS) 


Secretaria de Planificaciôn 
Estratégica 


Unidad de Calidad de la 
Educacion, Ministerio de 
Educación . Peru 


Latin American Center for 
Development 
Administration (CLAD) 


Mexican Institute for 
Competitiveness (IMCO) 


Center for Social 
Accountability and Studies 
in the Democratic 
Construction at CIESAS. 
Mexico 


New York University 


CLEAR Latin America 
Centre 


CLEAR Latin America 
Centre 


Centro de Investigacion y 
Docencia Económicas 
(CIDE), Mexico 


Red Latinoamericana de 
Evaluación (RELAC) 


World Bank 


Centro de Investigación y 
Docencia Económicas 
(CIDE), Mexico 


Centro de Investigación y 
Docencia Económicas 
(CIDE), Mexico 


Secretaría de Extensión 
/Universidad de la Punta - 
Argentina 


Centro de Investigación y 
Docencia Económicas 


Government 
(national) 


Government 
(national) 


Other 


Individual 
(consultant) 


NGO/CSO 


Academic/Re 
search 


Academic/Re 
search 


Academic/Re 
search 


Academic/Re 
search 


Academic/Re 
search 


Donor/Devel 
opment 
organisation 


Academic/Re 
search 


Academic/Re 
search 


Academic/Re 
search 


Academic/Re 
search 


Potential 
clients 


Potential 
clients 


Actual 
clients/partne 
rs 


Actual 
partners 


Actual clients 


Thought 
Leader 
(RAC) 


CLEAR staff 
& faculty 


CLEAR staff 
& faculty 


Actual 
partners 


Actual 
clients/partne 
rs 


Thought 
leaders 


Actual 
partners 


Actual 
partners 


Potential 
clients 


Host 
institution 
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TUDELA 
TRAVERSO, 
David Vera 


VELASCO, 
Ernesto 


VELASQUEZ 
VALDIVIA, Anibal 


VILLARRUEL, 
Juan Carlos 


M&E Unit 


Director General 


Director 


Journalist 


(CIDE), Mexico 


Ministerio de Desarrollo e 
Inclusión Social (MIDIS), 


Peru 


Civicus Consultants 
(Mexico and Peru) 


Dirección General de 
Seguimiento y Evaluación 
Despacho Viceministerial 
de Políticas y Evaluación 


Social, Peru 


Notimex Agencia de 
Noticias del Estado 
Mexicano, Mexico 


Government 
(national) 


Individual 
(consultant) 


Government 
(national) 


Other 


Actual clients 


Actual 
partners 


Actual clients 


Actual clients 
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F.6 South Asia 





AGRAWAL, Director Institute of Applied Government CLEAR 
Rashmi Manpower Research Competitor 
(IAMR) 
AIYAR, Yamini Senior Research Accountability Initiative, NGO RAC 
Fellow and Director India 
ARORA, Priya Events and CLEAR South Asia Research/ CLEAR SA 
Communications Academic 
Manager 
ASIM, Minahil Senior Research Punjab Monitoring & Government Client 
Associate Implementation Unit (sub- 
national) 
BANDARA, President Emeritus Sri Lankan Evaluation NGO Client/partner 
Nilanthi Association (SLEvA) 
BARNHARDT, Assistant Professor Indian Institute of Research/ CLEAR SA 
Sharon of Economics Management - Anmedabad Academic faculty 
(IIM-A) 
BHATTARAI, Principal Scientist, International Crops Research Client 
Madhu Economics Research Institute for the 
Semi-Arid Tropics 
(ICRISAT), India 
BUTEAU, Sharon Executive Director IFMR LEAD (formerly Research/ CLEAR SA 
known as IFMR Research) Academic faculty 
CHHIBBER, Ajay Director-General Independent Evaluation Government Client 
Office (IEO), Planning 
Commission, India 
DE SILVA, Soma Head, M&E Teaching Evaluation in NGO, RAC 
South Asia (TESA), Sri Research 
Lanka 
DHAR, Diva Program Director CLEAR South Asia Research/ CLEAR SA 
Academic 
ESCUETA, Maya Capacity Building CLEAR South Asia Research/ CLEAR SA 
Manager and Policy Academic 
Manager 
FLORETTA, John Deputy Director, J- CLEAR South Asia Research/ CLEAR SA 
PAL South Asia Academic 
GHAUS, Usman Teaching Fellow Lahore University of Research/ CLEAR SA 
Management Sciences Academic faculty 
GUPTA, Chief Economist USAID, India Development Client 
Madhumita organisation 
HAY, Katherine Senior Program Bill & Melinda Gates Development RAC 
Officer, Foundation, India organisation 
(foundation) 
JAIN, Tarun Assistant Professor Indian School of Business - Research/ CLEAR SA 
of Economics Hyderabad Academic faculty 
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JENKINS, 
Andrew 


JHA, Shreyasi 


Co-ordinator, 
Impact Assessment 
Unit 


M&E Specialist 


BRAC Research & 


Evaluation Division (RED), 


Bangladesh 
UN Women 


Development 
organisation 


Client 


Client/partner 





KANNAN, Harini Post-Doctoral J-PAL South Asia Research/ CLEAR SA 
Fellow Academic faculty 
KRISHNAN, Founder Adducense Innovations, Consultant Former 
Aparna India CLEAR SA, 
now faculty 
KRISHNAN, CV President Institute for Financial Research/ CLEAR SA 
Management and Academic Host 
Research (IFMR) India institution 
KUMAR, K. Shiva UNICEF — Retired Regional Advisory Development RAC 
Committee organisation 
LOMME, Roland Governance Advisor World Bank, India Development Partner 
organisation 
MIAN, Atif Board Director CERP (Center for Research/ CLEAR SA 
Economic Research in Academic Host 
Pakistan) - CLEAR Sub institution 
Centre 
MOHAMMED, Director, Interactive Research and Research/ Partner 
Shama Community Development Academic (faculty) 
Outreach 
MUKERJI, Executive Director, Abdul Latif Jameel Poverty  Research/ CLEAR SA 
Shobhini J-PAL South Asia Action Lab (J-PAL) South Academic 
Asia 
NAQVI, Syed Assistant Director, Department of Economic Government Client 
Zubair Husain Affairs, Ministry of Finance, 
Government of India 
PATTANAYAK, Unit Co-ordinator, Community of Evaluators NGO Client/Partne 
Sandip (CoE), South Asia r 
PURI, Jyotsna Deputy Executive 3ie NGO Partner 
Director and Head 
of Evaluation 
QURESHI, Assistant Chief Planning & Development Government Client 
Ahmed Waseem Development Department, AJK (sub- 
national) 
RIAZ, Maira Policy and Training CERP (Center for Research/ CLEAR SA 
Associate Economic Research in Academic 
Pakistan) 
SARWAR, Imran Co-founder Rabbt NGO Client 
SEN, Sree Capacity Building CLEAR South Asia Research/ CLEAR SA 
Associate Academic 
SHAHJAHAN, Country Director Innovations for Poverty NGO Partner 
Shoraez Action (IPA), Bangladesh 
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SHUKLA, Urmy Capacity Building CLEAR South Asia Research/ CLEAR SA 
Manager Academic 
STEVENSON, Associate Director CERP (Center for Research/ CLEAR SA 
Gemma Economic Research in Academic 
Pakistan) 
SUDARSHAN, Senior Research J-PAL South Asia and Research/ CLEAR SA 
Anant Manager (J-PAL Harvard University Academic (faculty) 
SA), Sustainability 
Science Fellow 
(Harvard) 
SUSHANT, Director, Research Breakthrough (NGO) NGO Client 
Leena 
TEJANI, Shirin Research Associate Centre for Research and Government Client 
Experiments for Action and  (sub- 
Policy (REAP), national) 
Government of Haryana, 
India 
TROYER Senior Research Harvard University, India Research Partner 
MOORE, Charity and Policy 
Manager, BCURE, 
IFMR LEAD, and 
Evidence for Policy 
Design (EPoD) 
WAHID, Mannan Associate Punjab Skills Development Government Client 
Wahid Fund (sub- 
national) 
ZAVERI, Sonal Treasurer Community of Evaluators, NGO Client/Partne 
South Asia r 
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Annex G List of Documents Consulted 


G.1 Documents sent by the CLEAR Secretariat or available online 


Relating to the start-up or design of CLEAR 


e CLEAR Initiative, “Charter”, current as of May 12, 2012, web page: 
http://www.theclearinitiative.org/Clear_about.html 


e “CLEAR Programme Description”, Web page: 
http://www. theclearinitiative.org/Clear_about.html 
Board Meeting Minutes 


e Minutes of the VC Meeting to Discuss the Proposal, December 19, 2008 
e Minutes of the Board Meeting, June 2, 2010 

e Board Meeting Minutes, July 29, 2010 

e Minutes of the Selection Committee Meeting, October 4, 2010 

e Minutes of the Board Meeting, December 9, 2010 

e Board Meeting, January 11, 2011 

e Minutes of the Board Meeting, March 30, 2011 

e Minutes of the Board Meeting, June 28, 2011 

e Draft Minutes of the CLEAR Board Meeting, November 22, 2011 

e Draft Minutes of the CLEAR Board Meeting, January 13, 2012 

e Draft Meeting Minutes, March 22, 2012 

e Draft Minutes of the CLEAR Board Meeting, (Videoconference), March 27, 2012 
e Draft Minutes of the CLEAR Board Meeting (Videoconference), May 3, 2012 
e Draft Minutes of the Board Meeting, May 30, 2012 

e Draft Minutes of the Board Meeting, July 3, 2012 

e Final Minutes of the Board Meeting, September 11, 2012 

e Draft Minutes of the Board Meeting, October 24, 2012 

e Draft Minutes of the Board Meeting, November 15, 2012 

e CLEAR Board Meeting Minutes, November 2012 

e Minutes of the Board Meeting February 20, 2013 

e Draft Minutes of the Board Meeting, March 13, 2013 

e Minutes of the Board Meeting, Tuesday April 30, 2013 

e Draft Minutes of the Board Meeting, Thursday, May 30, 2013 

e Final Minutes of the Board Meeting, Tuesday, July 2, 2013 


25 As of 28 May 2014 
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Final Minutes of the Board Meeting, Tuesday, July 23, 2013 

Draft - Brazil Selection Committee Meeting Minutes, August 29, 2013 
Draft Minutes of the Board Meeting, Thursday, September 26, 2013 
Minutes of the Board Meeting, Tuesday, November 19, 2013 
Minutes of the Board Meeting, Wednesday, January 15, 2014 

Board Meeting, February 19, 2013 

Board Meeting, February 27 


Report commissioned by CLEAR 


AccountAbility: Enhancing the Governance System of the CLEAR Initiative, Report 
presented to CLEAR, 2012. 


Legal documents setting out agreements between the World Bank Group and CLEAR host 
institutions 


Memorandum of Understanding between the Independent Evaluation Group, World Bank 
Group, and Ministry of Finance, China, and the Asia Pacific Finance and Development 
Centre: July 12, 2010 


Strategic documents 


“CLEAR Strategy (2013-2018): Development Anchored in Evidence, Learning, and Mutual 
Accountability”, September 2013 


CLEAR Old Theory of Change, used from 2011 to June 2013. 
http://www.theclearinitiative.org/Clear_about.html 


CLEAR Theory of Change, Updated June 2013. 
http://www.theclearinitiative.org/Clear_about.html 


“CLEAR Results Framework” based on the updated Theory of Change, current as of 
November 2013. 


“Building Blocks of CLEAR’s Capacity Development Strategy”, from February 1, 2013. Web 
page: http://www.theclearinitiative.org/Clear_about.html 


Financial documents 


CLEAR Budget from January 2011 to June 2012 (including financial contribution by CLEAR 
Donor by year) 


Grant Agreement with CESAG, October 2012 
Grant Agreement with CIDE, May 2013 
Grant Agreement with JPAL, July 2013 


World Bank, “Trust Fund Administration Agreement between the Swiss Confederation, 
represented by the Swiss Agency for Development and Cooperation (SDC), and the 
International Bank for Reconstruction and Development and the International Development 
Association concerning the Multi-Donor Trust Fund for Regional Centres for Learning on 
Evaluation and Results (CLEARY , 25/11/2011 


World Bank, Trust Fund Administration Agreement with the Inter-American Development 
Bank concerning the Multi-Donor Trust Fund for Regional Centres for Learning on 
Evaluation and Results (CLEAR) , 12/08//2011 
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World Bank, Trust Fund Administration Agreement between the Government of Australia 
and the International Bank for Reconstruction and Development and the International 
Development Association concerning the Multi-Donor Trust Fund for Regional Centres for 
Learning on Evaluation and Results (CLEAR) , 27/04/2011 


World Bank, Trust Fund Administration Agreement between the Kingdom of Belgium, acting 
through its Federal Public Service for Foreign Affairs, International Trade and Development, 
and the International Bank for Reconstruction and Development and the International 
Development Association concerning the Multi-Donor Trust Fund for Regional Centres for 
Learning on Evaluation and Results (CLEAR), 16/03/2011 


World Bank, Trust Fund Administration Agreement between the African Development Bank 
and the International Bank for Reconstruction and Development and the International 
Development Association concerning the Multi-Donor Trust Fund for Regional Centres for 
Learning on Evaluation and Results (CLEAR), 28/12/2009 


World Bank, Trust Fund Administration Agreement between the Asian Development Bank 
and the International Bank for Reconstruction and Development and the International 
Development Association concerning the Multi-Donor Trust Fund for Regional Centres for 
Learning on Evaluation and Results (CLEAR) , 19/01/2010 


Recent Amendments to the Grant Agreements for Board Members 


ADB, 23 August 2013 
AfDB, 24 June 2013 
AusAID, 24 June 2013 
Belgium, 24 June 2013 
DFID, 24 June 2013 
IADB, 28 August 2013, 
Rockefeller, 24 June 2013 
SDC, 24 June 2013 

SIDA , 24 June 2013 


Reports and Monitoring Tools 


CLEAR Initiative, State of Initiative Report, 2010 to June 2012. Web page: 
http://www.theclearinitiative.org/Clear_about.html 


Monitoring Tool Key 

Monitoring tool guidelines/instructions 
Template for CLEAR Annual Reports 
Other products 


Agenda from the Global Forum in Mexico, November 2013. http://www.clear- 
la.cide.edu/node/105 


CLEAR, Third Global Forum: Tunis 2013. 


CLEAR Global Tunis Forum: Strategic Framework Template: Planning Tool — Annexes. 
February 2013. 
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e Annex C: CLEAR Strategic Framework Template Planning Tool, Preparation for the CLEAR 
global Forum- Tunis, February 2013. 
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G.2 Documents concerning CLEAR Centres and Sub Centres 
Expressions of Interest and Proposals 
Anglophone Africa Centre 


e University of the Witwatersrand, Johannesburg: Expression of interest to be considered as 
a host institution for a regional centre for learning on evaluation and results (CLEAR), 
March 2010. 


e University of the Witwatersrand, Johannesburg: Proposal to host a Centre for Learning on 
Evaluation and Results at the University of the Witwatersrand’s Graduate School of Public 
and Development Management, 26 September 2010. 


South Asia Centre 

e CLEAR South Asia: “Requirements for Submitting Expressions of Interest.” 
Brazil 

e CLEAR at EESP-FGV: Project Proposal 

e CLEAR at EESP-FGV: Annex 4 CVs 

e CLEAR at EESP-FGV: Annex 1A Evidence and Sources of information for Section A1 
Francophone Africa Centre 

e CESAG Expression of Interest 

e CESAG Proposal to Host the CLEAR Centre in Francophone Africa 
Latin America Centre 

e Centre for Research and Teaching in Economics (CIDE): Expression of Interest 

e CIDE Proposal to Host the CLEAR Centre in Spanish-speaking Latin America 

e CIDE Proposal to Host the CLEAR Centre in Spanish-speaking Latin America: Annex 
General 

e Proposal Requirements and Site Visit Requirements 


e Timeline and Procedures for Short-listing and Selecting, CLEAR Centres in Francophone 
Africa and Latin America 


Strategy 
e Anglophone Africa Centre Project Document (December 2013) 
e CESAG CLEAR Project Document 


e WITS Strategic Planning. Web page: 
http://www.wits.ac.za/aboutwits/strategicplanningdivision/functions/15998/strategic_plannin 
g.html 


e CIDE Mexico Plan Estratégico: Web page: 
http://www.cide.edu/DIRECCION%20GENERAL/Plan_estrategico_CIDE.paf 


e CESAG Web page: 
http://www.cesag.sn/index.php?option=com_content&view=article&id=38&ltemid=1 84 
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Workplans, Narrative and Financial Reports 
e Anglophone Africa Overview Work Plan 
e Updated workplan from November 2011: J-Pal South Asia Centre 
e Brazil Centre EESP Budget 

Francophone Africa Centre 


e Annex 1: Simplified Financial Management Assessment Report: Data sheet and Risk 
Assessment 


e Annex: Results of Activities and Work Plan Year 1 
e Year One Budget for the CLEAR Francophone Africa Centre 


e CESAG: Rapport d’Opinion du Commissaire Aux Comptes, Exercice clos le 31 Décembre 
2008 


e CESAG: Rapport d’Opinion du Commissaire Aux Comptes, Etats financiers - Exercice clos 
le 31 Décembre 2009 


e Latin America Centre 
e CLEAR Budget CIDE 
Regional Centres procedures 
Anglophone Africa 
University of the Witwatersrand, Johannesburg. Recruitment, Selection and Appointment Policy. 13 
Feb 2012. 
Organisational Assessments 
e Mindfarm (2013) Organisational Assessment of CLEAR AA Centre 
Regional Centre Products 


e WITS: “African Monitoring and Evaluation Systems, workshop report: Collaborative 
Reflection and Learning Amongst Peers”, 26-29 March 2012. Web page: 
http://www.wits.ac.za/files/glgah_088716001350647108.paf 


e CLEAR and WITS: “Exploratory Case Studies: A Collection of Case studies facilitated by 
the CLEAR initiative - WITS”. Web page: 
http://www.wits.ac.za/files/glgah_826012001350647072.paf 


e The Presidency RSA (2012) Development Indicators 2012 Department of Performance, 
Monitoring and Evaluation 


e The Presidency RSA (2012) Mid-Term Review of the Priorities of Government 2012 
Department of Performance, Monitoring and Evaluation 


e The Presidency RSA (2012) National Evaluation Plan 2013-14 to 2015-16 Department of 
Performance, Monitoring and Evaluation 


e The Presidency RSA (2011) National Evaluation Policy Framework Department of 
Performance, Monitoring and Evaluation 


e WITS, Draft Study on the Demand for and Supply of Evaluation in Rwanda 


e WITS, Draft Study on the Demand for and Supply of Evaluation in Ethiopia, September 
2013 
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WITS, Draft Study on the Demand for and Supply of Evaluation in Zambia, October 2013 


CLEAR South Asia Centre: M&E Webinar Series: http://www.povertyactionlab.org/south- 
asia/clear-me-roundtable 


CIDE Evaluation Diploma (Nuevo Diplomado de Politicas Publicas y Evaluacion). Web 
page: http://www.clear-la.cide.edu/diplomado?language=en 


CIDE. List of publications. Web page: http://www.clear- 
la.cide.edu/publicaciones?language=en 


Francophone Africa Centre. “An Assessment of Demand for Monitoring, Evaluation, and 
Performance Management information and Services (2013).” Web page: 
http://www.theclearinitiative.org/center_africa.html 


Francophone Africa Centre. “Demand and Supply: Monitoring, Evaluation, and 
Performance Management Information and Services in Anglophone Sub-Saharan Africa: A 
Synthesis of Nine Studies (2013)”. Web page: 

http://www. theclearinitiative.org/center_africa.html 


DFID Annual Review and Summaries 


DFID. CLEAR Project Logframe: Revised February 2014. 


DFID. Annual Review of the Centres for Learning on Evaluation and Results (CLEAR)- 
Review undertaken Feb-March 2014. 


DFID. CLEAR Secretariat (FY14 Q1&Q2). A summary against the Secretariat work 
program. 


DFID. CLEAR — Anglophone Africa Centre FY14 Q18Q2. A summary against the 
Anglophone Africa Centre work program 


DFID. CLEAR- South Asia Centre FY14 Q1 &Q2. A summary against the South Asia 
Centre work program 


DFID. CLEAR- East Asia Centre FY14 Q1 &Q2. A summary against the East Asia Centre 
work program 


DFID. CLEAR- Francophone Africa Centre FY14 Q1 &Q2. A summary against the 
Francophone Africa Centre work program 


DFID. CLEAR- Latin America Centre FY14 Q1 &Q2. A summary against the Latin America 
Centre work program 


Relevant Background Articles 


African Development Bank (2013). Kingbo, Pierre-Joseph. How can we strengthen national 
evaluation systems (September 2013). Retrieved from: 


http://operationsevaluation.afdb.org/en/evaluations-publications/evaluation/evaluation- 
matters-how-can-we-strengthen-national-evaluation-systems-september-2013-347/ 


Avey, Paul and Michael Desch (forthcoming): “What Do Policymakers Want from Us? 
Results of a Survey of Current and Former Senior National Security Decision-makers”. 
International Studies Quarterly, Vol. 58, No. 4, December 2014. Retrieved on April 30 from: 
http://www3.nd.edu/~carnrank/PDFs/What%20D0%20Policymakers%20Want%20from%20 
Us_MC.pdf 
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Bezanson, Keith A. and Isenman, Paul (October 2012). Governance of New Global 
Partnerships: Challenges, Weaknesses and Lessons. Policy Paper 014. Centre for Global 
Development. Retrieved from: 

http://www.cgdev.org/files/1426627 file Bezanson Isenman FINAL.pdf 


Chouinard, Jill Anne (2013). The Case for Participatory Evaluation in an Era of 
Accountability. 


Clinton, Janet (2013). The True Impact of Evaluation: Motivation for ECB. 


ECDPM (2008). Capacity, Change and Performance: Insights and Implications for 
Development Cooperation (2008) Maastricht: ECDPM. 


Evaluation Capacity Development (2013). The Government Monitoring and Evaluation 
system in India: A Work in Progress, October 2013, ECD working paper series no 28 
Retrieved from: 

https://ieg.worldbankgroup.org/Data/reports/ecd_wp28 india me O.pdf?utm source=ME+i 
n+India%2FPBB+ECD+email&utm_ campaign=ECD+Email+External&utm medium=email 


Greenhill, R. and A. Prizzon and A. Rogerson (2013). The age of choice: developing 
countries in the new aid landscape. ODI Working Papers 364. Retrieved on April 15, 2014 
from: http://www.odi.org.uk/publications/7163-age-choice-developing-countries-new-aid- 
landscape 


Lusthaus, Charles and Christine Milton-Feasby. (2006). The Evaluation of Inter- 
Organizational Relationships in the Not-for-Profit Sector: Some Observations. Universalia, 
Montreal 


Mackay, K. (2007) on specific strategies for strengthening demand for M&E. 
http://www.worldbank.org/ieg/ecd/docs/How to build ME gov.paf 

Mackay, Keith (2007): How to Build M&E Systems to Support Better Government. World 
Bank/IEG. Retrieved on May 5 from: 

http://www.worldbank.org/ieq/ecd/docs/How to build ME gov.pdf 

Mayne (2009) Results Management: Can Results Evidence Gain a Foothold in the Public 


Sector? In: Reiper, O., Leeuw, F. and T. Ling (Eds.), 2009. The Evidence Book, 
Transaction Publishers 


Mayne (2009), and Kirkhart, K. (2000). Reconceptualizing evaluation use: An integrated 
theory of influence. In V. Caracelli 8 H. Preskill (Eds.), The expanding scope of evaluation 
use. New Directions for Evaluation, No. 88 (pp. 5-24). San Francisco, CA: Jossey-Bass 


Mayne, J. (2007). Best Practices in Results-Based Management: A Review of Experience - 
A Report for the United Nations Secretariat. New York: UN Secretariat 


Mayne, J. (forthcoming). Using Theories of Reach to Enhance Equity Considerations in 
Evaluation. Speaking Justice to Power: Ethical and Methodological Challenges for 
Evaluators. K. Forss and M. Marra, Eds, Transaction Publishers 


Mayne, John. (2014). Using Theories of Change in the Agriculture for Nutrition and Health 
(A4NH) CGIAR Research Program. Unpublished Draft, March 2014. 


NEPAD (2011). Final Draft African Consensus and Position on Development Effectiveness, 
September 2011. Fourth High Level Forum on Aid Effectiveness, Busan 


http://www.un.org/en/ecosoc/newfunct/pdf/luxembourg afr pos busan.paf 
OECD (2009). Development of Country Systems, Issues Brief no 1, 2009. OECD: Paris. 
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G.4 


OECD (2011): Perspective Note: The Enabling Environment for Capacity Development. 
Retrieved on May 5 from: http://www.oecd.org/development/governance- 
development/48315248.pdf 

OECD (2012). The Busan Partnership for Effective Development Cooperation 
http://www.oecd.org/dac/effectiveness/Busan%20partnership.pdf 


OPM (2009): Evaluation of the Paris Declaration: Thematic Study - Support to Statistical 
Capacity Building, Synthesis Report 


Pawson, R. and Tilley, N. (1997). Realistic Evaluation. London, Sage Publications 


Quinn Paton, M. (2010). Developmental Evaluation applying complexity concepts to 
enhance innovation and use. New York, NY: Guilford Press 


Sutcliffe and Court (2005). Evidence-Based Policymaking: What is it? How does it work? 
What relevance for Developing Countries? Overseas Development Institute, November 
2005. http:/Awww.odi.org.uk/sites/odi.org.uk/files/odi-assets/publications-opinion- 
files/3683.pdf 

UNDP (2013). Discussion Paper: Innovations in Monitoring and Evaluating Results 
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Annex H Roles and Responsibilities of the Secretariat 


CLEAR unit Types of support provided by the Secretariat 


CLEAR Designing and leading the process of centre selection 


overall Planning and implementing the global learning component of CLEAR, including 
organizing the annual Global Forum 


Providing operational oversight of CLEAR operations, including responsibility for the 
compilation and publication of CLEAR annual reports 


Communication, including the CLEAR global website 


Development of strategic and operational tools, such as the revised versions of the 
CLEAR ToC and Results Framework; various monitoring and reporting tools and 
templates; Document on “Building Blocks of CLEAR’s Capacity Development Strategy” 
(encouraging centres to think beyond training); 


Obtaining country clearances for the work to proceed in-country through the grants — 
coordinating with Ministries of Finance/Treasury, WB offices, etc 
CLEAR Board Liaising with individual Board members 
Preparing and managing Board meetings 
Writing position papers when requested by the Board 
Regional Reviewing RC contracts that fall under CLEAR, and handling a myriad of details that get 
Centres monitored, reviewed and reported. 


Acting as liaison between centres and various stakeholders in the respective regions, 
thereby helping to establish (potential) partnerships (e.g. with the local WB offices). 


Providing tailored guidance and technical support to individual centres, depending on 
their needs. Some examples are: 


Anglophone Africa: 


Technical guidance and international support to kick start the centre’s 
research/knowledge work on country M&E systems. 


Advice for structuring the relationship and identifying challenges and areas for support to 
various clients, including the Department of Performance Monitoring and Evaluation 
(DPME) of the South African Presidency 


Support for the transition in Centre’s Directors. 
Francophone Africa 


Design and review of the Centre’s first major study — demand assessment across three 
countries, resulting in recognition of the centre and improved visibility at conferences 
and network events. 


South Asia 


Feedback on work programme, particularly on the new sub-national level work that is 
being conducted and the grass-roots work that will be initiated with ASER. 


Latin America 
Provided peer review comments for various projects 
East Asia 


Helped establish the centre’s profile by including it in a high-level international seminar 
on evaluation requested by the Ministry of Finance 
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Annex | Interview Protocols 


Interview protocols for CLEAR Secretariat staff members, CLEAR Regional Centre staff, CLEAR 
clients, CLEAR Board Members. Regional Advisory Committee Members, CLEAR collaborators 
and stakeholders, representatives of Host Institutions, other M&E capacity development service 
providers, M&E and capacity development thought leaders are outlined below. They are based on, 
and aligned with, the evaluation questions outlined in Annex F. 


I.1 Interview Protocol: CLEAR Secretariat Staff 


ePact, represented by Universalia Management Group, a Canadian consulting firm, has been 
contracted by the UK Department for International Development (DFID) to conduct a Mid-Term 
Evaluation of the CLEAR Initiative. An important part of our data collection is to consult with 
CLEAR staff, clients and collaborators to understand their views and experiences with CLEAR to 
date. Please note that the focus of a Mid-Term Evaluation is to help CLEAR learn both about its 
successes as well as about areas that may require revision or further thinking. Please be assured 
that all information gathered in interviews will be treated confidentially. 


1. Background/Introduction 


If individual has not been interviewed before: Please briefly describe your role(s) and 
responsibilities within the CLEAR Secretariat. 


2. CLEAR progress and programme level performance at mid term 


Global level 


1. What internal and external factors have affected (positively or negatively) the establishment 
and functioning of the Secretariat to date? 


2. To what extent have actual global programming activities been congruent with CLEAR’s 
strategic documents and frameworks? If applicable, what factors have influenced changes 
to the originally envisaged portfolio of activities? 


3. What (types of) services, events, products offered by or through the Secretariat have been 
most successful? Least successful? 


4. What factors have led to the success (or lack thereof) of different types of global 
activities/services/products offered by CLEAR? (E.g. their thematic relevance; 
format/methodologies; actors, setting/location, timing) 


Regional level (to be focused on the Centre or Centres the respondent is familiar with) 


1. What factors have (positively or negatively) influenced the process of selecting and 
establishing the Regional Centres? 


2. To what extent have actual regional programming activities been congruent with the 
respective RCs’ strategic documents and frameworks? If applicable, what factors have 
influenced changes to the originally envisaged portfolio of activities? E.g. in terms of: 


e Types of services and distribution of budget allocations; 
e Types of clients served; 
e Levels of capacity built; 


e Thematic foci of products and services; 
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3. 


Geographic reach. 


What do you consider to be key achievements made by the Regional Centres to date”? 
Why? 


Relevance/Design 


1. 


To what extent does CLEAR constitute an appropriate response to the national, regional 
and global trends underpinning its creation? Is it keeping up with evolving trends? 


In what ways do CLEAR objectives and services take the specific and changing contexts in 
different countries and regions into account? Please give examples. 


. To what extent does the CLEAR design build upon and reflect (emerging) good practices in 


and current thinking on (M&E) capacity development, global networks, and global 
partnership programs? 


To what extent do CLEAR objectives and services take issues of inclusion and equality into 
account? (E.g. as regards the selection of issues/topics to address, identifying potential and 
actual target groups for services; or exploring the uses of M&E systems for/by different 
types of stakeholders). 


To what extent do CLEAR’s services complement or duplicate those offered by other 
providers? Are there examples of synergies generated between CLEAR and other actors? 


Looking forward 


1. 


How well is CLEAR positioned to contribute to developing and sharing global knowledge; 
and facilitating peer learning through the CLEAR network? Why? 


How well is CLEAR positioned to contribute to its intended regional learning outcomes? 
Why? 


Strengthening both supply of and demand for evidence in the region? 
Developing a critical mass of M&E professional expertise in the region? 
Innovation in and diversity of M&E approaches/methods in the Region? 


In your view, what would it mean for CLEAR to be sustainable at regional and global 
levels? What factors need to be in place to allow for sustainability? Which of these can 
CLEAR influence? 


3. Organisational Arrangements 


1. 


Are roles and responsibilities of key units within the CLEAR network (including the Board, 
Secretariat, Centres, sub-centres, host institutions, Advisory bodies) clear and appropriate? 
What, if any, issues require further clarification? 


. What aspects of the coordination, collaboration, problem-solving and decision-making of 


the CLEAR network and among CLEAR units are working/not working? Why? 


To what extent does the Secretariat have the human and other resources to effectively 
carry out its responsibilities? What are benefits and drawbacks of the current size and 
location of the Secretariat? 


Describe the added value of the CLEAR Board? 
How useful are the existing processes and tools for monitoring and reporting on CLEAR RC 


services and achievements for the Regional Centre in view of learning, planning, and 


decision making? Why? 





e-Pact 


73 


CLEAR Midterm Evaluation — Annexes 





6. What have been strengths and weaknesses of existing systems and processes for financial 
management for CLEAR, including receipt and flow of financial resources from donors? 


4. Looking ahead 
1. What mid-term corrective actions might be helpful to improve the effectiveness, relevance, 
sustainability and efficiency of the CLEAR programme at global and regional levels? 
2. Corrective actions conducted/initiated by 
e The CLEAR Secretariat 
e The Regional Centre 
e The CLEAR Board 
e Others (please specify). 


3. Please share any other comments or observations that may be relevant in the context of 
this evaluation. 


1.2 Interview Protocol: CLEAR Regional Centre Staff 


ePact represented by Universalia Management Group, a Canadian consulting firm, has been 
contracted by the UK Department of International Development (DFID) to conduct the Mid-Term 
Evaluation of the CLEAR initiative. An important part of our data collection is to consult with 
CLEAR staff, clients and collaborators to understand their views and experiences with CLEAR to 
date. Please note that the focus of a Mid-Term Evaluation is to help CLEAR learn both about its 
successes as well as about areas that may require revision or further thinking. Please be assured 
that all information gathered in interviews will be treated confidentially. 


1. Background/Introduction 


If individual has not been interviewed before: Please briefly describe your role(s) and 
responsibilities within/in relation to the CLEAR Regional Centre. 





e-Pact 74 


CLEAR Midterm Evaluation — Annexes 





2. CLEAR progress and programme level performance at mid term 


Global level 


1. 


What types of global events, products, services provided by CLEAR (through the 
Secretariat) have you participated in or used? 


How useful have these events, products, services been to you/the RC? What 
characteristics have contributed to their respective usefulness? (Including strengths and 
weaknesses of the CLEAR Secretariat) 


Regional level 


1. 


Do Or > A 


To what extent did the process of selecting and establishing the Regional Centre meet your 
expectations and needs? (E.g. as regards related processes, timelines, resources, support 
from CLEAR Secretariat and host institution). What, if any, are remaining gaps in the 
readiness of your RC? 


. What characterizes the portfolio of activities conducted by your Regional Centres to date? 


Were your original assumptions realistic as regards the composition and scope of the RCs 
portfolio? E.g. in terms of: 


Types of services and distribution of budget allocations; 

Types of clients served; 

Levels of capacity built; 

Thematic foci of products and services; 

Geographic reach. 

What do you consider to be the key achievements made by the RC to date? Why? 
What have been key factors supporting or hindering (progress towards) achievements? 
What, if any, unexpected results/achievements has the RC contributed to? How? 


In what ways do CLEAR objectives and services take the specific and changing contexts in 
different countries in your region into account? (Including stakeholder needs, as well as 
trends/thinking on M&E). Please give examples. 


To what extent do CLEAR objectives and services take issues of inclusion and equality into 
account? (E.g. as regards the selection of issues/topics to address, identifying potential and 
actual target groups for services; or exploring the uses of M&E systems for/by different 
types of stakeholders). 


To what extent do CLEAR’s services complement or duplicate those offered by other 
providers? Are there examples of synergies generated between CLEAR and other actors? 


How do the costs of CLEAR services compare to those of other providers? What is your 
assessment based on? 


Future progress and sustainability 


1. 


How well is CLEAR positioned to contribute to developing and sharing global knowledge; 
and facilitating peer learning through the CLEAR network? Why? 


How well is CLEAR positioned to contribute to its intended regional learning outcomes? 
Why? 


Strengthening both supply of and demand for evidence in the region? 


Developing a critical mass of M&E professional expertise in the region? 
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e Innovation in and diversity of M&E approaches/methods in the Region? 


3. In your view, what would it mean for CLEAR to be sustainable at regional and global 
levels? 


4. What factors need to be in place to allow for sustainability? Which of these can CLEAR 
influence/is already influencing? 


3. Organisational Arrangements 


1. Are roles and responsibilities of key units within the CLEAR network (including the Board, 
Secretariat, Centres, sub-centres, host institutions, Advisory bodies) clear and appropriate? 
Why? 


2. What aspects of the coordination, collaboration, problems-solving and decision-making of 
the CLEAR network and among CLEAR units are working/not working? Why? 


3. How effective is the CLEAR Secretariat in fulfilling its functions? Which of the Secretariat’s 
functions and/or services are most helpful for your Regional Centre? Why? 


4. How effective is the CLEAR Board in fulfilling its functions? 


. How effective is the Regional Advisory Body in providing guidance to the centres 
programming and implementation issues? Why? Why not? 


6. What are benefits and challenges deriving from the arrangement with your host institution? 
How has the relationship evolved since the RC was created? 


7. How useful are the existing processes and tools for monitoring and reporting on CLEAR RC 
services and achievements for the Regional Centre in view of learning, planning, and 
decision making? Why? 


4. Looking ahead 

1. What mid-term corrective actions might be helpful to improve the effectiveness, relevance, 

sustainability and efficiency of the CLEAR programme at global and regional levels? 

Corrective actions conducted/initiated by 

e The Regional Centre 

e The CLEAR Secretariat 

e The CLEAR Board 

e Others (please specify). 


2. Please share any other comments or observations that may be relevant in the context of 
this evaluation. 


L3 Interview Protocol: CLEAR Actual Clients 


ePact represented by Universalia Management Group, a Canadian consulting firm, has been 
contracted by the UK Department of International Development (DFID) to conduct the Mid-Term 
Evaluation of the CLEAR initiative. An important part of our data collection is to consult with 
CLEAR staff, clients and collaborators to understand their views and experiences with CLEAR to 
date. Please note that the focus of a Mid-Term Evaluation is to help CLEAR learn both about its 
successes as well as about areas that may require revision or further thinking. Please be assured 
that all information gathered in interviews will be treated confidentially. 
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1. Background/Introduction 


1. Please briefly describe your role(s) and responsibilities within your organisation. 
2. How did you learn about CLEAR and when? 
3. In your own words, what is CLEAR about? 


2. CLEAR progress and programme level performance at mid term 


Services and achievements 


1. How many and what types of services offered through CLEAR have you (or your 
organisation) used (e.g. events such as trainings, advisory services, knowledge products, 
professional exchange) 


a. How many individuals and what types (levels) have participated in/used CLEAR 
services? (Individuals, teams? Executive, managerial or technical/professional?) 


b. What were the respective thematic foci of these events? 
2. How useful have these events, products, services been to you/your organisation? 
In terms of strengthening knowledge/awareness (please specify topic) 
In terms of strengthening skills 
Improved consensus/team work 


> ® © 0 


Enhanced networks/strengthened coalitions 


New implementation know-how 


z= a 


Other (please specify) 


3. How relevant have been/are CLEAR’s services in view of existing needs and strategic 
priorities of your organisation/country? Please specify. 


4. To what extent do CLEAR objectives and services take issues of inclusion and equality into 
account? (E.g. do different types of stakeholder groups have equal access to its services? 
Are the topics and types of capacity development offered by CLEAR equally relevant to 
different actors with a stake in evaluation? Are you aware of attention being paid to how 
evolving M&E systems are being used for/by different stakeholders ?). 


5. What are remaining needs/gaps in your organisation? What is needed to address these? 
(Internally and/or with external assistance). 


6. In what ways, if at all, were CLEAR services that you have used tailored to your specific 
national/regional context? (E.g. in terms of methodology, language, key concepts). Please 
give examples. 


7. To what extent do CLEAR’s services complement or duplicate those offered by other 
providers? How do the costs of CLEAR services compare to those of other providers? What 
is your assessment based on? 


Future progress and sustainability 


1. How well is CLEAR positioned to contribute to its intended regional learning outcomes? 
Why? 


e Strengthening both supply of and demand for evidence in the region? 


e Developing a critical mass of M&E professional expertise in the region? 
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Innovation in and diversity of M&E approaches/methods in the Region? 
How relevant and realistic are these learning outcomes to existing needs in your 
country/region? 


In your view, what would it mean for CLEAR services to be sustainable? What factors need 
to be in place to allow for sustainability? 


3. Looking ahead 


1. 


What mid-term corrective actions might be helpful to improve the effectiveness, relevance, 
sustainability and efficiency of the CLEAR programme at global and regional levels? 


Corrective actions conducted/initiated by 


1.4 


The Regional Centre 
The CLEAR Secretariat 
The CLEAR Board 


Others (please specify). 


Please share any other comments or observations that may be relevant in the context of 
this evaluation. 


Interview Protocol: CLEAR Board Members 


ePact, represented by Universalia Management Group, a Canadian consulting firm, has been 
contracted by the UK Department for International Development (DFID) to conduct a Mid-Term 
Evaluation of the CLEAR Initiative. An important part of our data collection is to consult with 
CLEAR staff, clients and collaborators to understand their views and experiences with CLEAR to 
date. Please note that the focus of a Mid-Term Evaluation is to help CLEAR learn both about its 
successes as well as about areas that may require revision or further thinking. Please be assured 
that all information gathered in interviews will be treated confidentially. 


1. Background/Introduction 


Ifindividual has not been interviewed before 


1. 


2. 


Please briefly describe your role(s) and responsibilities within the CLEAR Secretariat. Since 
when have you been involved? 

What are the main reasons for your home agency’s interest in and support for the CLEAR 
initiative? Is your agency supporting any other similar global initiatives focus on 
strengthening M&E capacity? 


2. CLEAR progress and program level performance at mid term 


Global level 


1. 


In your view, what (types of) services, events, products offered by or through CLEAR at the 
global level have been most successful? Least successful? Why? How do you know? 


What factors have led to the success (or lack thereof) of different types of global 
activities/services/products offered by CLEAR? 
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Regional level 


1. 


2. 


What factors have (positively or negatively) influenced the process of selecting and 
establishing the Regional Centres? 


What do you consider to be key achievements made by the RCs to date? Why? 


Relevance/Design 


1. 


To what extent does CLEAR constitute an appropriate response to the national, regional 
and global trends underpinning its creation? Is it keeping up with evolving trends? How? 


. To what extent does the CLEAR design build upon and reflect (emerging) good practices in 


and current thinking on (M&E) capacity development, global networks, and global 
partnership programs? 


. To what extent do CLEAR objectives and services take issues of inclusion and equality into 


account? (E.g. as regards the selection of issues/topics to address, identifying potential and 
actual target groups for services; or exploring the uses of M&E systems for/by different 
types of stakeholders). 


To what extent do CLEAR’s services complement or duplicate those offered by other 
providers? Are there examples of synergies generated between CLEAR and other actors? 


Looking forward 


1. 


How well is CLEAR positioned to contribute to developing and sharing global knowledge; 
and facilitating peer learning through the CLEAR network? Why? 


How well is CLEAR positioned to contribute to its intended regional learning outcomes? 
Why? 

Regional outcomes: Strengthening both supply of and demand for evidence in the region; 
developing a critical mass of M&E professional expertise in the region; Innovation in and 
diversity of M&E approaches/methods in the Region? 


In your view, what would it mean for CLEAR to be sustainable at regional and global 
levels? What factors need to be in place to allow for sustainability? Which of these can 
CLEAR influence? 


3. Organizational Arrangements 


1. 


Are roles and responsibilities of key units within the CLEAR network (including the Board, 
Secretariat, Centres, sub-centres, host institutions, Advisory bodies) clear and appropriate? 
What, if any, issues require further clarification? 


What aspects of the coordination, collaboration, problems-solving and decision-making of 
the CLEAR network and among CLEAR units are working/not working? Why? 


How effective is the CLEAR Secretariat in fulfilling its functions? What are benefits and 
drawbacks of the current size and location of the Secretariat? 


How effective is the CLEAR Board in fulfilling its functions? What are benefits and 
drawbacks of the current size, composition and method of operation of the Board? 


How useful are the existing processes and tools for monitoring and reporting on CLEAR 
services and achievements in view of learning, planning, and decision making? Why? 


. What have been strengths and weaknesses of existing systems and processes for financial 


management for CLEAR, including receipt and flow of financial resources from donors? 
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4. Looking ahead 


1. 


What mid-term corrective actions might be helpful to improve the effectiveness, relevance, 
sustainability and efficiency of the CLEAR program at global and regional levels? 


Corrective actions conducted/initiated by 


2. 


The CLEAR Secretariat 
The Regional Centre 
The CLEAR Board 
Others (please specify). 


What factors are likely to determine the continued interest and support of your 
organization/agency to CLEAR? 


Please share any other comments or observations that may be relevant in the 
context of this evaluation 


1.5 


Interview Protocol: CLEAR Collaborators 


ePact, represented by Universalia Management Group, a Canadian consulting firm, has been 
contracted by the UK Department for International Development (DFID) to conduct a Mid-Term 
Evaluation of the CLEAR Initiative. An important part of our data collection is to consult with 
CLEAR staff, clients and collaborators to understand their views and experiences with CLEAR to 
date. Please note that the focus of a Mid-Term Evaluation is to help CLEAR learn both about its 
successes as well as about areas that may require revision or further thinking. Please be assured 
that all information gathered in interviews will be treated confidentially. 


1. Background/Introduction 


= N 


Please briefly describe your role(s) and responsibilities within your organization. 
How did you learn about CLEAR and when? 
In your own words, what is CLEAR about? 


What was your organization’s key motivation for partnering with CLEAR? 


2. CLEAR progress and program level performance at mid term 


Services and achievements 


1. 


How many and what types of services has your organization developed and/or offered 
jointly with CLEAR? (e.g. events such as trainings, advisory services, knowledge products, 
professional exchange) 


How many individuals and what types (levels) have participated in/used these services? 
(Individuals, teams? Executive, managerial or technical/professional?) 


What were the respective thematic foci of these events? 


How useful have these events, products, services been to the respective clients? How do 
you know? 


In terms of strengthening knowledge/awareness (please specify topic) 
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In terms of strengthening skills 

Improved consensus/team work 

Enhanced networks/strengthened coalitions 
New implementation know-how 

Other (please specify) 


How relevant have been/are CLEAR's services in view of existing needs and strategic 
priorities of the targeted organizations/of the country? Please specify. 


. What are remaining needs/gaps in targeted organizations/in your country? What is needed 


to address these? 


In what ways were the services provided by your organization and CLEAR tailored to the 
specific national/regional context(s) that they took place in? (E.g. in terms of methodology, 
language, key concepts). Please give examples. 


To what extent did these services complement or duplicate those offered by other 
providers? How do the costs of these services compare to those of other providers? What 
is your assessment based on? 


To what extent do CLEAR objectives and services take issues of inclusion and equality into 
account? (E.g. as regards the selection of issues/topics to address, identifying potential and 
actual target groups for services; or exploring the uses of M&E systems for/by different 
types of stakeholders). 


Future progress and sustainability 


1. 


How well is CLEAR positioned to contribute to its intended regional learning outcomes? 
Why? 

Regional learning outcomes: Strengthening both supply of and demand for evidence in the 
region; developing a critical mass of M&E professional expertise in the region; Innovation in 
and diversity of M&E approaches/methods in the Region? 


In your view, what would it mean for services such as the ones provided by your 
organization and CLEAR to be sustainable? What factors need to be in place to allow for 
sustainability? 


3. Looking ahead 


1. 


What mid-term corrective actions might be helpful to improve the effectiveness, relevance, 
sustainability and efficiency of the CLEAR program at the regional level? 


What, if any, mid-term corrective actions could further improve your collaboration with 
CLEAR? 


Please share any other comments or observations that may be relevant in the context of 
this evaluation. 
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1.6 Interview Protocol: Other Evaluation Capacity Building Providers 
(CLEAR Competitors)?* 


ePact, represented by Universalia Management Group, a Canadian consulting firm, has been 
contracted by the UK Department for International Development (DFID) to conduct a Mid-Term 
Evaluation of the CLEAR Initiative. An important part of our data collection is to consult with 
CLEAR staff, clients, collaborators and competitors to understand their views and experiences with 
CLEAR to date. Please note that the focus of a Mid-Term Evaluation is to help CLEAR learn both 
about its successes as well as about areas that may require revision or further thinking. Please be 
assured that all information gathered in interviews will be treated confidentially. 


1. Background/Introduction 
1. Please briefly describe your organization's work and approach in view of evaluation 


capacity building (ECB) (e.g. geographic foci; thematic foci; methodological foci) 


2. How did you become aware of CLEAR and how? What is your relationship with CLEAR 
now? Has it evolved over time? 


3. In your view, what are the key characteristics of the CLEAR initiative? 
2. CLEAR progress and program level performance at mid term 
1. What (types of) services, events, or products offered by/ through CLEAR at the global and 
regional levels are you aware of? 
2. Which, if any, have you used? (e.g. publications) Why? With what impression/benefit? 


Relevance/Design 


1. To what extent does CLEAR constitute an appropriate response to the national, regional 
and global trends underpinning its creation? 


2. To what extent does the CLEAR design build upon and reflect (emerging) good practices in 
and current thinking on (M&E) capacity development? 


3. To what extent do CLEAR's services complement or duplicate those offered by your 
organization/other providers? 


4. Are there examples of synergies generated between CLEAR and your organization/other 
actors? 

5. To what extent do CLEAR objectives and services take issues of inclusion and equality into 
account? (E.g. as regards the selection of issues/topics to address, identifying potential and 
actual target groups for services; or exploring the uses of M&E systems for/by different 
types of stakeholders). 


Looking forward 


1. How well is CLEAR positioned to contribute to developing and sharing global knowledge; 
and facilitating peer learning through the CLEAR network? Why? 


26 Organizations/providers who have not (yet) collaborated with CLEAR, i.e. different from CLEAR 
collaborators. 
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2. How well is CLEAR positioned to contribute to its intended regional learning outcomes? 
Why? 
Regional outcomes: Strengthening both supply of and demand for evidence in the region; 
developing a critical mass of M&E professional expertise in the region; Innovation in and 
diversity of M&E approaches/methods in the Region? 


3. In your view, what would it mean for CLEAR to be sustainable at regional and global 
levels? What factors need to be in place to allow for sustainability? Which of these can 
CLEAR influence? 


3. Looking ahead 

1. What mid-term corrective actions might be helpful to improve the effectiveness, relevance, 

sustainability and efficiency of the CLEAR program at global and regional levels? 

Corrective actions conducted/initiated by 

e The CLEAR Secretariat 

e The Regional Centre 

e The CLEAR Board 

e Others (please specify) 


2. Please share any other comments or observations that may be relevant in the context of 
this evaluation. 


I.7 Interview Protocol: Representatives of Host Institutions for 
Regional Centres 


ePact, represented by Universalia Management Group, a Canadian consulting firm, has been 
contracted by the UK Department for International Development (DFID) to conduct a Mid-Term 
Evaluation of the CLEAR Initiative. An important part of our data collection is to consult with 
CLEAR staff, clients and collaborators to understand their views and experiences with CLEAR to 
date. Please note that the focus of a Mid-Term Evaluation is to help CLEAR learn both about its 
successes as well as about areas that may require revision or further thinking. Please be assured 
that all information gathered in interviews will be treated confidentially. 


1. Background/Introduction 


Ifindividual has not been interviewed before: 


1. Please briefly describe your role(s) and responsibilities within your institution. What is your 
relationship with the CLEAR Regional Centre and related responsibilities (if any)? 


2. What were the main reasons for your institution's interest in the CLEAR initiative (including 
any links to your institution's priorities and/or any similar M&E capacity strengthening 
initiatives)? 


2. CLEAR progress and program level performance at mid term 


Global level 


1. What, if any, global events, products, services provided by CLEAR (through the Secretariat) 
have you participated in or used? 
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2. 


How useful have these events, products, services been to you/your organization? What 
characteristics have contributed to their respective usefulness? 


Regional level 


1. 


g Oi P 


To what extent did the process of selecting and establishing the Regional Centre/Sub- 
centre meet your expectations and needs? (E.g. as regards related processes, timelines, 
and resources). What, if any, are remaining gaps in the readiness of the RC that your 
organization is hosting? 


What benefits/opportunities did your institution anticipate from hosting the Regional Centre? 
To what extent have these expectations been met/are likely to be met? 


What characterizes the portfolio of activities conducted by the Regional Centre hosted by 
your organization to date? Were its original assumptions realistic as regards the 
composition and scope of the RC’s portfolio? E.g. in terms of: 


Types of services and distribution of budget allocations; 

Types of clients served; 

Levels of capacity built; 

Thematic foci of products and services; 

Geographic reach. 

What do you consider to be the key achievements made by the RC to date? Why? 
What have been key factors supporting or hindering (progress towards) achievements? 


In what ways do CLEAR objectives and services take the specific and changing contexts in 
different countries in your region into account? (Including stakeholder needs, as well as 
trends/thinking on M&E). Please give examples. 


To what extent do CLEAR objectives and services take issues of inclusion and equality into 
account? (E.g. as regards the selection of issues/topics to address, identifying potential and 
actual target groups for services; or exploring the uses of M&E systems for/by different 
types of stakeholders). 


To what extent do CLEAR’s services complement or duplicate those offered by other 
providers? (Including your organization, if/as applicable). Are there examples of synergies 
generated between CLEAR and other actors (including your organization)? 


How do the costs of CLEAR services compare to those of other providers (including other 
parts of your organization)? What is your assessment based on? 


Future progress and sustainability 


1. 


How well is CLEAR positioned to contribute to developing and sharing global knowledge; 
and facilitating peer learning through the CLEAR network? Why? 


How well is CLEAR positioned to contribute to its intended regional learning outcomes? 
Why? 


Strengthening both supply of and demand for evidence in the region 
Developing a critical mass of M&E professional expertise in the region 
Innovation in and diversity of M&E approaches/methods in the Region? 


In your view, what would it mean for CLEAR to be sustainable at regional and global 
levels? 
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What factors need to be in place to allow for sustainability? Which of these can CLEAR 
influence/is already influencing? 


3. Organizational Arrangements 


1. 


Are roles and responsibilities of key units within the CLEAR network (including the Board, 
Secretariat, Centres, Sub-centres, host institutions, Advisory bodies) clear and appropriate? 
Why? 


. What aspects of the coordination, collaboration, problems-solving and decision-making of 


the CLEAR network and among CLEAR units are working/not working? Why? (Including 
collaboration between Regional Centre and host institution). 


What are benefits and challenges deriving from the arrangement between the RC and your 
organization as the host institution? How has the relationship evolved since the RC was 
created? 


4. Looking ahead 


1. 


What mid-term corrective actions might be helpful to improve the effectiveness, relevance, 
sustainability and efficiency of the CLEAR program at global and regional levels? 


Corrective actions conducted/initiated by 


1.8 


The Regional Centre/Sub-Centre 

The CLEAR Secretariat 

The CLEAR Board 

Your organization/institution 

Please share any other comments or observations that may be relevant in the context of 
this evaluation. 


Interview Protocol: Potential CLEAR Clients 


ePact represented by Universalia Management Group, a Canadian consulting firm, has been 
contracted by the UK Department of International Development (DFID) to conduct the Mid-Term 
Evaluation of the CLEAR initiative. An important part of our data collection is to consult with 
CLEAR staff, (actual and potential) clients, collaborators and competitors to understand their views 
of and experiences with CLEAR to date. Please note that the focus of a Mid-Term Evaluation is to 
help CLEAR learn both about its successes as well as about areas that may require revision or 
further thinking. Please be assured that all information gathered in interviews will be treated 
confidentially. 


Y 
2. 
3. 


Please briefly describe your role(s) and responsibilities within your organisation. 
Presuming that you are aware of CLEAR, how did you learn about CLEAR and when? 


In your own words, what is CLEAR about? What types of products and services does 
CLEAR offer? To whom? 


How relevant are the following areas that CLEAR works in for your organization? Why? 


a. Helping to strengthening knowledge/awareness of development evaluation, the 
generation and use of evidence to inform decision making 





e-Pact 


85 


CLEAR Midterm Evaluation — Annexes 





10. 


1.9 


b. Strengthening skills related to the generation and/or use of evidence (through 
evaluation) 


c. Strengthening the organization in terms of internal consensus/team work towards 
generating evidence through evaluation and using this evidence 


d. Enhanced networks/strengthened coalitions working towards the generation/use of 
evidence for decision making 


e. New implementation know-how as regards the design, management, or use of 
evaluation to generate evidence 
f. Other (please specify) 


Have you used the services of other providers to help your organization in one or more of 
these areas? Why did you choose this provider/these providers? 


Have you ever considered working with CLEAR? If yes, on what issues? If not, why not? 
Under what conditions would you consider using the services of CLEAR? 


To what extent do CLEAR’s services complement or duplicate those offered by other 
providers? How do the costs of CLEAR services compare to those of other providers? What 
is your assessment based on? 


Are there any key gaps in your organisation for which you have, until now, been unable to 
find external assistance? (l.e. issues not offered by CLEAR or any other service provider). 


. CLEAR'S intention is to contribute to the following regional learning outcomes. How relevant 


and realistic are these learning outcomes to existing needs in your country/region? 
Strengthening both supply of and demand for evidence in the region? 

Developing a critical mass of M&E professional expertise in the region? 

Innovation in and diversity of M&E approaches/methods in the Region? 

Please share any other comments or observations that may be relevant in the context of 
this evaluation. 


Interview Protocol: Regional Advisory Committee Members 


ePact, represented by Universalia Management Group, a Canadian consulting firm, has been 
contracted by the UK Department for International Development (DFID) to conduct a Mid-Term 
Evaluation of the CLEAR Initiative. An important part of our data collection is to consult with 
CLEAR staff, clients and collaborators to understand their views and experiences with CLEAR to 
date. Please note that the focus of a Mid-Term Evaluation is to help CLEAR learn both about its 
successes as well as about areas that may require revision or further thinking. Please be assured 
that all information gathered in interviews will be treated confidentially. 


1. Background/Introduction 


1. 


Please briefly introduce yourself, in particular your connection to the issue of Evaluation 
Capacity Building. 


Since when have you been a member of the Regional Advisory Committee (RAC)? How 
were you approached to join the RAC, and by whom? 
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2. CLEAR progress and program level performance at mid term 


Global level 


1. 


What, if any, global events, products, services provided by CLEAR (through the Secretariat) 
are you aware of, or have you participated in/ used? 


2. How useful have these events, products, services been to you? What characteristics have 

contributed to their respective usefulness? 
Regional level 

1. What characterizes the portfolio of activities conducted by the Regional Centre/Sub-centre 
to date? E.g. in terms of: 

e Types of services and distribution of budget allocations 

e Types of clients served 

e Levels of capacity built 

e Thematic foci of products and services 

e Geographic reach 

2. What do you consider to be the key achievements made by the RC/SC to date? Why? 

3. What have been key factors supporting or hindering (progress towards) achievements? 

4. In what ways do CLEAR objectives and services take the specific and changing contexts in 
different countries in your region into account? (Including stakeholder needs, as well as 
trends/thinking on M&E). Please give examples. 

5. To what extent do CLEAR’s services complement or duplicate those offered by other 
providers? Are there examples of synergies generated between CLEAR and other actors? 

6. In what ways, if any, do you consider the work of CLEAR to be innovative, or to promote 
innovation? Why/how? 

7. To what extent do CLEAR objectives and services take issues of inclusion and equality into 


account? (E.g. as regards the selection of issues/topics to address, identifying potential and 
actual target groups for services; or exploring the uses of M&E systems for/by different 
types of stakeholders). 


Future progress and sustainability 


1. 


How well is CLEAR positioned to contribute to developing and sharing global knowledge; 
and facilitating peer learning through the CLEAR network? Why? 


How well is CLEAR positioned to contribute to its intended regional learning outcomes? 
Why? 


Regional Learning Outcomes: Strengthening both supply of and demand for evidence in the 
region; Developing a critical mass of M&E professional expertise in the region; Innovation in 
and diversity of M&E approaches/methods in the Region. 


In your view, what would it mean for CLEAR to be sustainable at regional and global 
levels? 

What factors need to be in place to allow for sustainability? Which of these can CLEAR 
influence/is already influencing? 
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3. Organizational Arrangements 


1. 


Are roles and responsibilities of the RAC vis-a-vis the Regional Centre/Sub-Centre clear? 
Are the roles and responsibilities of other CLEAR units clear (Board, Secretariat, host 
institutions)? 

How effective is the Regional Advisory Body in providing guidance to the Centre’s 
programming and implementation issues? Why? Why not? 


How useful are the existing processes and tools for monitoring and reporting on CLEAR 
Regional (Sub) Centre services and achievements in view of learning, planning, and 
decision making? Why? 


4. Looking ahead 


1. 


What mid-term corrective actions might be helpful to improve the effectiveness, relevance, 
sustainability and efficiency of the CLEAR program at global and regional levels? 


Corrective actions conducted/initiated by 


The Regional Centre/Sub-Centre 
The RAC 

The CLEAR Secretariat 

The CLEAR Board 

Others (please specify) 


Please share any other comments or observations that may be relevant in the context of 
this evaluation. 
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Annex J CLEAR Charter 


Attachment 1 
(Provided for informational purposes only) 


CHARTER 
REGIONAL CENTERS FOR LEARNING ON EVALUATION AND RESULTS (CLEAR) 


(As current on May 12, 2012, subject to amendment) 


Rationale and Objectives 


Management for results requires governments to have the capacity to monitor, measure, and 
evaluate the performance of public programs. However, these capacities are extremely weak in 
many governments, imperilling the effective management of public programs and therefore the 
achievement of results. The Evaluation of the Implementation of the Paris Declaration (2008) and 
the subsequent Accra Agenda for Action (2008) both reinforce the need to improve partner 
countries’ capacities for results measurement and evaluation. At the same time, the supply of 
appropriate, demand-driven, and cost-effective capacity building services is limited. 


Description of the Program 


The Regional Centres for Results-Based Management (RBM) and Monitoring and Evaluation 
(M&E) Capacity Program (henceforth referred to as “the Program”) is a collaborative effort among 
donors and partner countries to strengthen the M&E capacity of partner countries for RBM. This 
goal will be achieved by supporting a demand-driven, relevant, and cost-effective supply of 
regional RBM and M&E capacity building services to government and others to enable them to 
conduct their M&E functions. The Program is expected to create a situation in which governments 
and civil society can access capacity development services through regional institutions located in 
partner countries, rather than relying on developed country institutions. Thus, a major benefit of 
this Program is expected to be building the capacity to build RBM and M&E capacity in partner 
countries. 


The specific objectives of the Program are to: 


e Select and support regional academic/training centres in sub-Saharan Africa, East Asia & 
Pacific, Latin America, and South Asia to provide demand-driven capacity building services 
in RBM and M&E on a regional basis (the “Centres”). Key principles underlying the 
selection of the Centres will be: 


e Demonstration of demand for services from public sector agencies and civil society 
e Articulation of a demand-based program of activities, with a business plan 


e Provide a multi-regional forum for exchange of ideas, knowledge, and information on RBM 
and M&E systems and methods to enable practitioners (government and professionals) to 
learn from each other. 

Program Activities 


The Program will engage in three major sets of activities: 
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1. Select and support (through cash funding and technical assistance) regional 
academic/training institutions to establish Centres for RBM and M&E capacity building 
services. The Asia-pacific Finance and Development Centre (AFDC) in China, which 
already has an existing program with the Independent Evaluation Group (IEG) of the World 
Bank, will be folded into the Program. The funding and technical assistance provided to the 
selected Centres will enable the Centres to strengthen their relationship with government 
agencies and civil society and provide a broad array of demand-based, relevant, and cost- 
effective regional capacity building services, which may include: 


Training 


Based on an assessment of demand for training from specific agencies in specific 
countries: 


e Develop and deliver customized short- and long-term courses/seminars on RBM and M&E 
for specific audiences, including government officials, parliamentarians, civil society 
organizations, and donor staff 


e Customize modules for mainstreaming into existing degree, diploma, or certificate courses 
and for training of trainers 


e Develop and implement various modalities of learning, including e-learning and distance 
learning 


Other Knowledge Services 


e Establish or engage with a community of practice (CoP) for application of RBM and 
evaluation knowledge (e.g., a knowledge clearinghouse, an expert resource Center) 


e Based on consultations with govt. agencies and civil society, develop and provide M&E- 
related knowledge services, such as organization of regional conferences and hosting 
websites and other fora for M&E networks 


Advisory Services 


e Provide research and advisory services through engagement in evaluations (and therefore 
learning by doing) and development of M&E systems with government agencies 


2. The second component of the program will encompass multi-regional knowledge-sharing 
and peer-to-peer learning activities (including conferences, seminars, and training events) 
focusing on a range of topics: government M&E and accountability systems, social 
monitoring, organizational performance management systems, and so on. It will also 
include technical work carried out in support of the program. 


3. The third component of the program will support operations of the secretariat of the 
Program (the “Secretariat”) and management and administration of the program: support to 
the governing board of the Program (the “Board”) for partnership activities and 
administration of the trust fund established to support the program. 


The Centres for Results-Based Management (RBM) and Monitoring and Evaluation (M&E) 
Capacity is envisioned as a two-phase Program, with the first phase encompassing the first five 
years, during which centres are expected to move toward self-financing. The Board may extend 
the Program into phase two, depending on a number of factors, including demand for the 
Program’s activities and the success of the Program as evidenced through an evaluation. 


Governance of the Program 


The governance structure of the Program will comprise the Board and the Secretariat at the 
Program level and regional advisory committees at the Centre level (one for each region: East Asia 
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& Pacific, Africa, Latin America, and South Asia) (the “Regional Advisory Committees”). The terms 
of reference for each of these bodies are attached in Annex A to this Charter. 


Selection of the Centres 


The candidate Centres will be screened and shortlisted against a set of criteria agreed upon by the 
Board. Key criteria will include the potential Centre’s assessment of, and response to, the specific 
demand for RBM and M&E capacity building services in its region. The Secretariat will short-list 
the institutions and the Board will make the final selection decisions. 


Funding 


The Program will rely on direct cash funding from donors, as well as in-kind support provided by 
the partner countries, institutions, and the Bank. Each donor contributing cash-funding to the 
Program will enter into a trust fund agreement (“Administration Agreement”) with the Bank. Donor 
funds will be managed and administered by the Secretariat, based on the Administration 
Agreements and Board decisions. The majority of these funds will be recipient-executed and will 
be allocated directly to the selected Centres. A small portion of the funds will be used for multi- 
regional activities and Secretariat and Board functions. 


Monitoring and Evaluation 


The Program will be subject to regular monitoring and reviews by the Board. The selected 
Centres will submit annual reports to the Board through the Secretariat, and the Secretariat will 
compile an Annual Program Report summarizing Program financial and operational performance 
(which will also be submitted to all donors to the Program). The Board will also commission 
independent mid-term and final audits and evaluations. The Board will ensure the independence 
of the audits and evaluations by appointing external and independent consultants or firms. The 
Secretariat will only provide administrative support to the evaluation as necessary, such as 
providing temporary office space as needed by the consultants and assigning a staff member to 
facilitate the work of the evaluation team by such things as (a) providing key documents, (b) 
facilitating contacts with Program constituents and members of the governing body, and (c) 
facilitating access to local videoconference facilities. The M&E function will also comply with the 
World Bank’s fiduciary and reporting requirements. 


Communications and Outreach 


The Secretariat will maintain a Program website which will provide information regarding the 
Program, including lessons learned, and serve as a platform exchanging information among the 
Centres. The Secretariat will also play an active role in knowledge-dissemination regarding the 
Program through conferences, workshops, and meetings with potential partners. 


Collaboration with Other Programs Related to RBM and M&E 

Through the Centres, the Program will collaborate, as appropriate, with a number of initiatives that 
are addressing critical aspects of RBM to further the impact of its work. These include, for 
example, statistical capacity building, M&E communities of practice and networks, and impact 
evaluation initiatives, such as the SIE. 

Miscellaneous 


This Charter will become effective upon its adoption by the first of the Donors to enter into an 
Administration Agreement with the Bank on the basis of the form of the Charter attached to such 
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Administration Agreement. Amendments to this Charter may be made by the Board in accordance 
with the terms of this Charter and will become applicable to the Administration Agreements as such 
amendments take effect without further need to amend the Administration Agreements, provided 
that such amendments do not conflict with Bank policies and procedures and provided further that 
in the event of any conflict with any Administration Agreement, the terms of such Administration 
Agreement will prevail. Any amendments to the Charter will be directed through the Secretariat for 
review and confirmation prior to consideration by the Board. 


ANNEX A 
TERMS OF REFERENCE FOR THE GOVERNING BOARD 


The Board constitutes the main decision making body for the Program. The Board will set broad 
policy directions and goals for the Program and consider the Secretariat’s recommendations 
regarding the work program and annual budget. It will also oversee the activities of the Program. 


Specifically, the Board will be responsible for: 
1. Providing strategic direction to the Program 


Defining the TORs for the Regional Advisory Committees 


e Deciding on the selection process and selection criteria for the Regional Centres (the 
“Centres”) 


e Creating ad-hoc expert panels to provide advice to the Board regarding the Program (e.g., 
defining the criteria and processes for selecting the regional centres) 


e Selecting the Centres and approving their respective Trust Fund-funded work programs 
and business plans (including confirmation of the Regional Advisory Committee members) 


e Approving the annual budget of the Program and distributions among Centres and 
multiregional activities 


e Setting membership criteria and procedures of the expanded Board and potentially making 
adjustments to such criteria, procedures or overall Board composition 


e Amending the Charter by consensus, subject to the terms of the Charter, the Administration 
Agreements and Bank policies and procedures 


e Determining the scope and direction of the Program beyond the initial five years 


e Developing and approving a results-framework for the Program within the first year of its 
operation. 


2. Reviewing Program progress and effectiveness 


e Conducting yearly reviews of the Program’s and the Centres’ accomplishments and work- 
program based on annual reports (covering financial and operational performance); the 
Secretariat will provide the reports. 


e Commissioning independent evaluations and audits to determine the Program’s 
effectiveness and impact. The Board will ensure the independence of the audits and 
evaluations by appointing external and independent evaluators and auditors. 
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3. Communications and outreach 
e Communicating regarding the Program with external stakeholders and audiences 


e Raising funding for the Program. 
Size and Composition 


The Board will comprise members representing donors contributing US$500 thousand or more per 
year; one rotating seat for those contributing less than US$500 thousand; IEG; and the Bank’s two 
regional VPUs (for the first two years for regional donor representation). The organizations will 
designate their representatives. Donors contributing less than US$500 thousand per year will 
decide who among them will be their representative on the Board. It is expected that Board 
membership will later be expanded to include additional members representing senior government 
officials from partner countries (one from each region in which the selected Centres are located, 
based on biennial rotation among participating countries), after the Board establishes a procedure 
for government representation. 


Appointment and Tenure of Members 


Board members will serve for a period of two years, which may be extended, based on compliance 
with agreed membership criteria. The non-donor members of the Board will be invited by the 
Board for a similar period of a two-year renewable term. The Board will be chaired by the Bank. 


It is anticipated that membership on the Board may need to change as more donors and more 
regions enter the Program. If the need arises, the Board will discuss the procedures to expand or 
change its membership, either to replace an existing member who is no longer able to participate, 
to help address new issues, or to represent new stakeholders. All changes to Board membership 
will be guided by an agreed-upon criteria and procedures for selecting members of the Board. 


Management of Business 


The Board will meet at least once a year, but business may also be carried out virtually and in sub- 
groups. The Board's conclusions and decisions will be reached by consensus. The Chair may 
invite participation from the non-Board members of the Regional Advisory Committees, as well as 
from the Centres. The Board members will not be compensated for their time through the 
Program’s funds. Donors will bear the costs of their members’ Board-related activities. However, 
travel costs for the non-donor members and other invitees will be borne by the Program. 


The Board’s operations will be supported by a Secretariat, housed at the Bank. 
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TERMS OF REFERENCE FOR THE REGIONAL ADVISORY COMMITTEES 


Each Regional Center will appoint a Regional Advisory Committee (RAC), the membership of 
which will be subject to approval by the Board. The RACs will serve in an advisory capacity to the 
Centers for developing and implementing a strategic and high-quality regional program. Upon 
request, RACs will also provide advice to the Board regarding the Program’s strategic directions, 
particularly as they pertain to regional issues. 


Specifically, the RACs will be responsible for: 
1. Providing strategic guidance and support 


e Provide strategic guidance regarding the Centre’s overall vision and direction, focusing on 
its relevance for critical development issues in the region, innovation, and technical 
excellence 


e Network on behalf of the Centres for both non-financial and financial support and to help 
position and establish its reputation and profile as a leader in the field 


e Facilitate key connections and contacts in the region 


e Assist the Centre in establishing a sustainable model for future growth and continued 
relevance 


2. Providing advice on capacity development strategy 


e Providing nuanced and well-informed knowledge on the demands and needs for RBM and 
M&E capacities in their regions and/or specific countries, within their socio-political contexts 


e Providing advice regarding the overall content of the work program, focusing on demand, 
supply, and influence strategies 


e Guiding the Centres regarding effective methods for developing and delivering capacity 
building services in the region 


3. Providing guidance on implementation approaches 


e Advising Centres regarding a strategic assessment of their approaches to capacity 
development, including approaches to collaborating with partners and networks 


e Helping to identify opportunities for partnerships with other institutions in the region 


e Helping to identify individuals or organizations that could help design and deliver capacity 
building in the region 


e Providing advice and feedback regarding Centres’ performance 


Each Centre will determine the relative emphasis it places on the areas of advice noted above, 
based on its overall strategy, needs, and level of development. 


Size and Composition 


Each Centre will determine the size and composition of its own RAC (a maximum of seven 
individuals recommended). The RAC members should collectively provide the credibility, capacity, 
and recognized profile to be able take on responsibilities outlined in this terms of reference. In 
addition, the composition of the RACs should reflect diversity and the ability to function cohesively 
as a group. The RAC members will serve in their professional capacities and not as 
representatives of their organizations. The Chair of the Advisory Committee will be determined by 
the Centre. 
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Appointment and Tenure of Members 


The Centers will propose their RAC members for approval by the Board, as part of their work 
program plans. The term of the RAC members will be determined by the Centres. The Centres will 
also determine whether RAC membership is renewable. 


The RACs will meet face-to-face at least annually, with additional meetings being determined on 
the basis of the Centres’ needs. Members’ travel costs will be borne by the Centres’ from “Centres’ 
Development” the portion of their grants. Any bilateral, multilateral, and foundation financial donor 
organizations’ representatives on the advisory committees will fund their own costs. The RAC 
members will not be compensated for their time. If RAC members are invited for CLEAR Board 
meetings, the travel costs will be paid for by the Program funds held centrally by the Secretariat. 


Reporting and Accountability 


RAC business may also be carried out virtually and in sub-groups. Minutes of the RACs’ meetings 
and RAC’s recommendations will be recorded by the Centres and submitted to the CLEAR Board 
with their annual reports. 


TERMS OF REFERENCE FOR THE SECRETARIAT 


The Secretariat will be responsible for the day-to-day administration of the Program, based on the 
Board’s guidance and decisions. It will report directly to the Board’s Chairperson. The Secretariat 
will be housed at the Independent Evaluation Group (IEG) of the Bank, and staffed with Bank staff 
and secondees from donor and partner governments. 


The specific areas of responsibilities will be as follows: 
1. Administration of the Program, based on the Board’s decisions 
e Organizing Board meetings and supporting other Board activities 


e Preparing and submitting an annual report to the Board (and to all donors) on the 
Program’s financial and operational performance, including each Centre’s annual report 


e Monitoring the Program’s activities and work program 

e Providing and coordinating technical assistance to the Centres 

e Reviewing and confirming proposed amendments to the Charter 

e Maintaining and distributing clear records of amendments to the Charter 
e Other activities required by the Board 

2. Managing the selection of the Centres 


e Preparing a selection process and criteria for the Centres (‘Selection Process and Criteria”) 
and presenting it to the Board for their discussion, modification and approval 


e Managing the preparatory activities (i.e., diagnosis of regional M&E supply institutions, 
analysis of how the current initiative relates to other programs such as statistical capacity 
building, MfDR CoPs, and 3IE) to get the Program activities started 


e Managing the solicitation, screening, and reviewing of proposals for selecting the Centres, 
in accordance with the “Selection Process and Criteria” approved by the Board. 


3. Organizing and delivering the global component of the Program 


e Designing and delivering the global aspects of the Program (training, conferences, 
knowledge-exchange programs, knowledge products and curriculum materials) 
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e Organizing annual events with all Centres. 
4. Coordinating communications and outreach 
e Managing partner relations 


e Maintaining effective relationships with other potential donors, partner county governments, 
research and academic institutions, evaluation associations, communities of practitioners, 
local academics and NGOs, and other stakeholders 


e Managing the general internal and external communications 
e Maintaining a Program website. 


The Secretariat will be housed in IEG for the first phase (first five years) of the Program. The Bank 
will contribute facilities and staff time to the Secretariat. After the first five years, the location and 
staffing of the Secretariat will be reviewed to assess whether it could be moved to a partner 
country location. 
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Annex K Analysis of Board Agendas and decisions taken 


Approximate 
number of 


meetings that 
covered the 
item 


Type of decision taken 





Global 
Programming 


Program 
oversight 


Governance 


Selection of 
Regional 
Centres 


Expansion 


Board Process 


Finance 


Structure, 
Network 


18 


le 


12 


11 


e Decision about the organisation of the Forums (definition of the 
agenda, work program, location of the forum, component of the 
forum, who can participate, selection of the speakers invited, 
expected results/outcomes) 


e Decision regarding constituent's demand should take place with 
the Centres in order to assess needs and create capacity building 
strategies and approaches. 

e Revision and complement of the structure of the annual report 


e Decision about the proposal for the Clear Initiative to submit to the 
World Bank (1st Board meeting) 

e Decision about the first and second Chair of the Board 

e Clarification the vision of the Program 

e Decision about a communication and branding strategy 

e Definition of the TORs of the RACs 

e Composition and definition of the role of the different task forces 

e Delegation of work to the secretariat 

e Implementation of the recommendation of the Accountability 
report. The option 1: Status quo plus was chosen and no change to 
the Board composition. The discussion of the report lead to: 

e Development of CLEAR global value (or business case) 

e Refining the overall strategy 

e Definition of the competencies for a new Director of the 
Anglophone African centre 


e Selection of the Anglophone Africa centre, the Latin America 
centre (Mexico) and the Brazil Centre. 


e Approval to expand a centre to Brazil 
e Definition of a expansion strategy 
e Inclusion of a pacific centre in the expansion strategy 


e Agreement and approval of Board “Decision Papers” to help focus 
on key discussions and decisions 
e Approval about the agenda of the Board meetings 


e Decision to provide $5,000 per institution to conduct the demand 
assessment 

e Decision to reduce the annual contribution for the full Board 
membership to $500K 

e Allocation of resource to the global component of the program 

e Decision about expending a grant to a centre 


none 
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Annex L Comments on CLEAR Results Framework 


COn 





Highest level 
outcomes to which 
CLEAR contributes 


Stakeholders use 
evidence in making 
decisions for 
improved 
development results 


Higher-level 
outcome to which 
CLEAR contributes 


Strengthened 
context-specific 
M&E systems and 
practices 


1. Strategic clients 
(those who are 
important for 
influencing changes in 
M&E systems) report 
they are using 
evidence for decision 
making. 


1. Strategic clients 
report that CLEAR 
regional Centres have 
contributed to the 
strengthening of M&E 
systems 


2. Strategic clients 
report that they apply 
what they have 
learned from CLEAR. 


1. By 2018, 70 per cent of 
strategic clients and 
stakeholders surveyed 
report increased use of 
evidence in decision 
making. 


1. By 2018, an external 
evaluation commissioned 
by the Board indicates that 
centres have contributed to 
strengthening of M&E 
systems. 


2. By 2018, an external 
evaluation commissioned 
by the Board indicates that 
at least 70 per cent of 
CLEAR clients are using 
the knowledge, skills, or 
information they gained to 
raise evaluation practice. 


In absence of a more elaborated Theory of Change it is not evident 
whether, how, and to what extent CLEAR would be able to link any 
progress against this indicator and target to CLEAR contributions. In 
other words, the assumed transition between the higher level outcome 
and this highest level outcome is problematic. 


The findings of a future external evaluation do not constitute a useful 
target that centres or other CLEAR units could work with in a meaningful 
way. 


The current indicators that focus on client perceptions avoid the question 
of what success or progress (e.g. in terms of strengthening M&E 
systems) will look like. However, for centres to be able to 
measure/capture their respective contributions they will need to capture 
data on changes in specific actors' behaviours demonstrating the 
beginnings of actively utilised capacity. Similarly, they need to define 
what major milestones along the path towards this fully realised capacity 
would look like. 


Among those served by the centres one may, for example, look for 
evidence of changes in: self-empowerment; space for error and 
experimentation; positive deviance; flow of active learning; iterative 
adaptation; expecting adaptive management. On the part of donors, one 
could expect strategies that support such developments. 


The role of the global (overall) CLEAR results framework could be to 
outline possible types of indicators that the centres may want to measure 
in this regard, while the centres themselves would be in charge of setting 
specific targets/identifying specific milestones relevant in their countries 
or regions. 
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Fresuts [indicators Targets [Comments O OO O OOOO O OOOO OOU U OO 








Regional Leaming [AAA a 


CLEAR outcomes 


for Regional 
Learning 


Enhanced enabling 
environment and 
strengthened 
demand 


Strengthened 
organisational 
capacity to produce 
and use evidence 


Critical mass of 
professional 
expertise developed 


Innovation in M&E 


1. Centres’ range of 
capacity development 
increases from 
baseline to 
encompass different 
objectives, modalities, 
and M&E 
methodologies to 
address M&E 
capacity at different 
levels 


2. Centres’ work with 
strategic clients and 
stakeholders for their 
region 

3. Centres provide 
services in countries 
throughout their 
region 

4. Centres lead M&E 
innovation within their 
regions 

5. Clients indicate that 
the Centres’ services 
and activities are of 
high quality. 


1. By their third year, 
centres demonstrate in 
their work plans the 
capacity to address a range 
of M&E topics and 
methodologies (increase 
from baseline). 


2. By their third year, 
centres demonstrate in 
their annual work plan the 
capacity to offer capacity 
building through a variety of 
modalities aimed at 
different capacity objectives 
(increase from baseline). 


3. By their third year, at 
least 50 per cent of centre 
projects engage clients 
from outside of the centre’s 
home country. 


4. By their 3rd year, at least 
80 per cent of service 
clients score the quality of 
service as a 4 or higher (on 
a five-point scale). 


5.By 2018, an external 
evaluation indicates that 
the centres are functioning 
well with respect to their 
strategic plans and 
objectives 


The current indicators and milestones do not measure the envisaged 
outcomes, but progress in the set-up/functionality of the regional centres. 
The underlying assumption is that if the centres are functioning within the 
agreed upon parameters, then CLEAR will achieve results that contribute 
to the envisaged outcomes. This assumption is problematic. 


None of the indicators/targets helps to define (and measure progress in 
relation to) the enabling environment. As CLEAR evolves, it would make 
sense for each centre to define relevant progress markers indicating what 
changes in the enabling environment would look like. In some contexts 
this may relate to changes in legislation, or institutionalization of M&E, in 
other contexts it may be something else. At the global level CLEAR may 
merely want to monitor evidence of progress against these region- 
specific indicators, rather than identifying the same targets across 
centres. 


The current targets are activity focused. This is not helpful in terms of 
trying to capture progress against outcomes. 


Indicator/target # 1& 2: The mere fact of the centres offering a variety of 
M&E topics through a variety of modalities does not provide evidence of 
progress toward the envisaged outcomes. For example, some activities, 
whether training or TA, may be gateway activities to build larger 
partnerships. Others may be expansive strategies to get CLEAR 
recognition across a number of countries. 


Indicator/target # 3: The number/percentage of projects engaging clients 
from other countries does not provide insights into the strategic 
relevance, depth of work, and different client partnerships taking place 
across the region. Also, the new centre in Brazil is not likely to work 
beyond the country level, so this indicator/target would not apply. One 
possible approach to track progress in this regard can be to 
continue/repeat a Network Analysis similar to the one conducted for this 
evaluation to capture changes in the positioning of the CLEAR centres in 
their regions over time. 


Indicator/target #4: The quality of service is not equivalent with innovation 
in M&E. Also, while assessment scores from training and other activities 
are an important source of information, they do not provide a sufficient 
basis for assessing the quality of CLEAR’s work. CLEAR still needs to 
define whether ‘quality’ in this regard means utility for participants, or 
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CLEAR 
programme-level 
outputs 


Regional centres 
established and 
functional 


Global Approach 


Outcomes of global 
learning 


CLEAR global 
knowledge 


Peer-learning 
through the network 


1. Centres 
established in host 
institutions 


2. Centres functioning 
well against their 
plans 


CLEAR is recognised 
for excellence and 
innovation in 
supporting regional 
Centres. 


2. CLEAR Centres 
and their constituents 
participate in the 
CLEAR global 
programmes and 


1. By 2012, five centres 
selected and operational 
(original target was four). 


2. By 2018, an external 
evaluation indicates that 
the centres are functioning 
well with respect to their 
strategic plans and 
objectives 


3. By 2018, centres’ per 
cent of revenue-generating 
activities and programmes 
increase from baseline 
(targets will vary centre to 
centre). 


1. By 2018, centre directors 
and staff report that they 
have been able to apply 
knowledge gained from 
other Regional Centres 
through the CLEAR 
initiative. 


2. By 2018, the Regional 
Centres choose to 
continuing sharing 
knowledge and expertise 
through a global network. 


(also) ‘in line with international good practices’. 


The CLEAR Annual Program Review for FY2012/13 conducted by DfID 
recommended that the Secretariat establish a clear comparable basis for 
monitoring Centre progress. It noted that “ An empirical and impartial 
analysis of the establishment of the Centres is difficult to make due to a 
lack of specification of ‘establishment’ benchmarks and the lack of 
comparable data to assess progress. “ 


A related question for consideration by the CLEAR Board is what 
minimum information donors require that would assure them that the 
different centres are ‘functional’ — while acknowledging their different 
contexts and capacities. 


The DfID CLEAR Annual Program Review for FY2012/13 remarked that: 
“consideration should be given to whether public goods are collective 
outputs from CLEAR or from the Centres and Secretariat separately. 
Indicators to be adjusted accordingly.” 


The results statement, indicators and targets are based on the underlying 
(but still unproven and vague) assumption that ifthe CLEAR initiative is 
functional and recognised as a source of excellence, that it will then 
contribute to its envisaged development results. 
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Fresuts TT [Targets Comments 


peer-learning events. 3. By 2018, a survey of 


strategic clients and 
3. CLEAR develops a le 
high-quality global stakeholders indicate that 


at least 80 per cent 
knowledge products = 
and CD approaches. recognise the CLEAR 


global brand as a source of 
excellence and innovation 
in M&E. 
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Annex M What is evidence and how is it used 


What constitutes evidence? 


There are differing views on what is considered to be evidence. For some, evidence-informed 
policy making and decision making implies the need to use the strongest possible scientific 
evidence available, such as from randomised control trials. Others argue that evidence can be 
generated by any systematic effort to bring empirical information based on observation or 
experiment to the management table ” — for example through monitoring and evaluation. © 


When and how is evidence incorporated into policy making? 


Sutcliffe and Court (2005) note 
that policy processes usually 
involve different stages, from 
agenda-setting to formulation to 
implementation. While evidence 
has the potential to influence 
the policymaking process at 
each stage, different types of 
evidence and different 
mechanisms may be required 
for different stages.*° 


As is illustrated in the sidebar, 
the timeliness of evidence 
being made available to 
relevant actors is among the 
key factors determining 
whether and how evidence 
influences decision making. 


Factors that influence the use of evidence in policy/decision 
making 


Mayne (2009) argues that evidence is most likely to be considered in 
decision making if it is simultaneously: 


Relevant, i.e. if it pertains to an issue or decision to be addressed, AND 


Timely, i.e. made available at times when it can be considered (e.g. 
when decisions about something have to be made), AND 


Understandable, i.e. digestible for someone who may not be immersed 
in a particular topic and/or versed in the discourse of scientific research, 
AND 


Reliable and credible in the eyes of potential users and their 
constituents, e.g. based on the fact that the evidence is seen to be 
accurate and based on methodologically sound processes; AND 


If there is interest in results information by those involved. 29 


What other factors influence policy making? 


The good news is that evidence can matter. The bad news is that it often does not. 


131 


27 Mayne (2009) points out that evidence is thus distinct from any type of information, in that it focuses on 
empirical information ‘based on observation and experiment, not on theory’ (Concise Oxford Dictionary). 


28 According to the UK Cabinet Office, evidence is constituted by previous policy evaluations, but also by: 
expert knowledge; published research; existing research; stakeholder consultations; the Internet; outcomes 
from consultations; costings of policy options; output from economic and statistical modeling’. Source: 
Cabinet Office, quoted in Sutcliffe and Court (2005). 


29 Sutcliffe and Court (2005) describe similar factors that determine whether evidence is likely to be useful to 
policymakers: its quality, accuracy and objectivity; its credibility; relevance (timely, topical, with policy 
implications); and Practicalities (e.g. is it presented in a form that makes it easy to translate it into policy. 


30 This emphasises that not each type of evidence is suitable for each type of decision, i.e. evidence 
generated by M&E is relevant for certain types of decisions, but not for others. 


31 Julius Court speaking on ‘The political context in developing countries’, at Does Evidence Matter? ODI 
meeting series. Quoted in: Sutcliffe and Court (2005). 
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There is extensive literature around the observation that at each stage of the policy or planning 
cycle many different factors and types of information (other than evidence) may influence decision 
making.*? Often, even if relevant, timely, understandable, reliable and credible information is 
available, it is not used to inform decisions (see sidebar). This is the case in both individual and 
institutional decision making. 


This implies that initiatives aiming to strengthen the use of evidence in decision making should, 
ideally, i) be based on a thorough understanding of the various factors that are likely to influence 
decision-making processes in a particular context; ii) clarify whether and how each of these factors 
will be addressed through the 
initiative; and/or iii) if — as is likely 
— the initiative cannot address all 


of these key factors, identify 

which factors it can address, and Those involved do not ‘trust’ empirical information, believing rather 

with what likely results and in their own experience and knowledge, or information derived from 
hearsay, beliefs etc. 


Factors that may inhibit the use of available evidence in 
policy/decision making** 


limitations. 

Davies (2004:4-7)% describes The information does not fit their ideological base. 

seven major factors other than The information is inconvenient for power struggles, budgets, or 
evidence, which inform and garcon 

influence policy making. The implications arising from the information challenge current 


practices and require more willpower to change than there is. 
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Paul Avye and Michael Desch (forthcoming)* note that consulted policy makers expressed a need 
for arguments or theories (i.e. frameworks for making sense of the world they have to operate in) 
over the generation of specific evidence. 





32 For an overview and bibliography, please see Mayne (2009) and Sutcliffe and Court (2005). 
33 After Mayne (2009). 
34 Quoted in: Sutcliffe and Court (2005). 


35 Paul Avey and Michael Desch (forthcoming): What Do Policymakers Want from Us? Results of a Survey of 
Current and Former Senior National Security Decision-makers. International Studies Quarterly, Vol. 58, No. 
4, December 2014. Retrieved on April 30 from: 
http://www3.nd.edu/~carnrank/PDFs/What%20D0%20Policymakers%20Want%20from%20Us_MC.pdf 
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Annex N Demand and Supply in selected countries that 
CLEAR centres work in 


The following tables capture some elements of the enabling environments in the countries that the 
CLEAR centres and their affiliates are located in (with the exception of East Asia, for which there 
were no data).“*The tables do not attempt to comprehensively describe the contexts, but 
summarise some key characteristics and developments relevant to the evaluation, in particular as 
regards various types of incentives for M&E and RBM. 


Mackay (2007) argues that demand for M&E (i.e. the degree to which evidence generated through 
M&E is requested and used)?” can be strengthened through the following types of influences: 


e ‘Sticks’ — prods or penalties for organisations or individuals who fail to take performance 
and M&E seriously; common tools that act as ‘sticks’ include legal and policy frameworks 
and related guidelines. 


e ‘Carrots’ — ways of providing positive encouragement and rewards for conducting M&E and 
utilising the findings. These include, for example, public recognition or financial incentives 
to individuals or organisations. 


e ‘Sermons’ — high-level statements of endorsement and advocacy concerning the 
importance of M&E. These also include efforts to raise awareness of M&E and to explain to 
government officials (or other actors) “what’s in it for them”. 





36 In the case of Francophone Africa, the table also summarises information on other countries that were 
covered by the demand study carried out by the CLEAR FA centre. 


37 As noted in Chapter 4 on design, it is, in our view, problematic to assume that requesting evidence also 
means that available evidence is being used. However, the strategies suggested by Mackay may well be 
effective for strengthening both requests for, and use of evidence generated through M&E systems. These 
tables are a work in progress and may be revised in the final draft. 
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N.1 Anglophone Africa3* 


Other countries 





“Carrots” 


“Sticks” 


No wide culture that supports 
M&E and PM: According to 
Wits studies, lack of a sense 
of ownership of the M/E 
system is explained by a 
culture of “malicious 
compliance” and little interest 
in information use. (D&S) P. 
38. 


The 1996 Constitution 
established the foundations 
for M&E in South Africa in its 
description of the obligations 
of the state and the 
accountability requirements 
of the Executive. (D&S) 


The Municipal Systems Act 
of 2000 requires 
municipalities to develop their 
own PM systems, specifying 
targets, publishing reports, 
and reviewing performance. 
P.36 


Policy framework: The 
government-wide M&E policy 
framework of 2007 describes 
the roles and responsibilities 
of each government agency 


No wide culture that supports 
M&E and PM. 


In spite of its many M&E 
efforts, Kenya is reported to 
have a weak M&E culture, 
with M&E considered a 
policing tool, rather than a 
tool to improve delivery, 
efficiency and effectiveness. 
(D&S) P.20 


The government will approve 
an evaluation policy to set 
minimum M& standards 
(D&S) p. 9 


The Economic Recovery 
Strategy for Wealth and 
Employment Creation that 
the government use M&E to 
track its policies, programs, 
and projects. Under it, the 
government created the 
National Integrated 
Monitoring and Evaluation 
System and the Monitoring 
and Evaluation Directorate 
(MED) to lead and 
coordinate the system. 


P. 19. Lacks adequate 


No wide culture that supports 
M&E and PM. 


Ministries, departments, and 
agencies are required to 
monitor and evaluate the 
performance of their programs 


and projects periodically. They 


receive incentives or 
penalties, in response to their 
performance against targets. 
(D&S) p.15 


In ministries, there are policy 
planning and M&E 
departments that assess the 
performance of government 
interventions. Many of these 
units lack adequate capacity 
to perform their roles 
effectively and have minimal 
evaluation functions (D&S) 
Pale 


None of the governments 
covered in these studies is 
described as having 
established a government- 
wide culture that supports 
M&E and PM and the use of 
M&E and PM findings. (D&S) 
PS 


M&E is often viewed as a 
control and policing tool (D&S) 
P. © 


In Ghana, Kenya, Rwanda, 
South Africa, and Uganda, 
their constitutions established 
the foundation for M&E and 
PM. (Source: Demand and 
supply ... 2013. P. 9) 


Three countries (Kenya, 

South Africa, and Uganda) 
have developed national 
evaluation policies. (D&S) P. 9 


Donors play a large role in 
generating demand for M&E 
services. (D&S) P 10 


In all countries 
nongovernmental entities and 
the media serve a watchdog 
function, in addition to 
Parliaments. (D&S) P. 12 


38 Main sources: Demand and Supply: Monitoring, Evaluation, and Performance Management Information and Services in Anglophone Sub-Saharan 
Africa, CLEAR, 2013; Proposal to Host a Centre for Learning on evaluation and results (CLEAR) for Africa. 
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involved in the national 
monitoring and evaluation 
system. (D&S) p. 37 


In 2009 introduced the 
Framework for Managing 
Programme Performance 
Information, the South 
African Statistical Quality 
Framework and the 
“Improving Government 
Performance: Our Approach” 
policy document. 


In 2011: National Evaluation 
Policy Framework (D&S) P. 
67 


The outcomes-based 
approach has become the 
overarching framework for 
M&E in South Africa. (D&S) 
EST 


The Green Paper on National 
Strategic Planning highlights 
the link between planning 
and M&E. p. 37 


Challenges aligning 
legislation and policies. 
(D&S) P. 37 


Starting in the mid-1980s, 
donor investments in the 
non-profit sector and their 
accompanying accountability 
requirements laid the 
foundation for programs. 
(D&S) P. 38 


The Department of 
Performance Monitoring and 
Evaluation in South Africa 
was established on 1 


Kenya 
coordination. (D&S) P.23 


The government of Kenya 
has a draft national 
evaluation policy awaiting 
Cabinet approval. (D&S) P. 
19 


The Local Government Act, 
Public Service Commission 
Act, Public Procurement and 
Disposal Act, and the 
Constitution of 2010 create 
demand for M&E, with the 
last explicitly requiring 
accountability and 
transparency from public 
institutions. (D&S) P 19, 


Program-based budgeting, 
led by Business 
Development Services in the 
Ministry of Finance, 
demands M&E of public 
sector programs and 
projects, as well as the 
development of sector 
targets and indicators. (D&S) 


The Anti-Corruption and 
Economic Crimes Act (2003), 
implemented by the Ministry 
of Justice, Cohesion and 
Constitutional Affairs, calls 
for monitoring public sector 
service delivery 
achievement. A number of 
other budget, financial and 
audit laws and regulations 
further enhance the 
government's results focus. 
(D&S) 


Evaluation associations are 


listed for all the countries. 
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“Sermons” 


January 2010 has produced 
a range of frameworks and 
undertaken some 
implementation activities that 
demand a ‘big-bang’ 
implementation across 
government. Proposal, p. 5 


Provincial level: respond 
changes. At this level there is 
certainly demand for greater 
clarity around the legal 
reporting requirements from 
Treasury and the new 
(unlegislated) requirements 
from the Presidency’s 
Department of Performance 
Monitoring and Evaluation 
(PM&E). proposal p. 6 


Fifteen Year Review of 
Government: 
Recommendations for 
improving the government's 
implementation capacity 
included improving the M&E 
system, giving those with 
oversight responsibilities 
powers of sanction, and 
emphasizing achievement of 
outcomes over compliance. 
(D&S) 


Kenya 


Demand for improved M&E 
capacities, transparency and 
accountability came from: 
donors, Kenya’s citizens, the 
media and Parliament. (D&S) 
Al 


Media plays a watchdog role. 
(D&S) P. 21 


NGOs also have institution- 
specific M&E systems that 
are separate from each other 
and from the national M&E 
system. (D&S) P. 23 


Constituency Development 
Fund Act and the Local 
Government Authority Act 
require that CSOs be 
involved in the project 
identification and monitoring 
processes. The nature of 
their involvement is 
unspecified. (D&S) P.24 


High-level government 
officials have spoken publicly 
about the need to 
demonstrate results. (D&S) P 
20 


Kenyan Human Rights 
Commission argues that the 
new constitution embodies 
the “fundamental principles 
necessary for the realisation 
of accountability within the 
Government of Kenya.” 
Source: Proposal, p. 5 


The Kenyan M&E Directorate 
does not have sufficient 
institutional knowledge of the 


The last strategy promotes 
transparent and accountable 
governance and places 
greater emphasis on results- 


based management (D&S) (p. 


15) 

The government of Ghana 
recognizes that undertaking 
an assessment of its M&E 
resources would help it know 
where to target its efforts to 
strengthen its M&E capacities 
(D&S) p.15 


Uganda has prioritized M&E 
and PM capacity building in 
their national development 
plans. (D&S) P. 10 


In many countries the 
government has built a 
government-wide M&E 
system to assess its progress 
against poverty reduction 
strategies and national 
development plans. (D&S) P. 
11 
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current M&E terrain to 
respond to this new context. 
Proposal, p. 5 


M&E entities Department of 
Performance, Monitoring 
and Evaluation in the 
Presidency established in 
2010. It has no legislative 
backing and currently exists 
at the prerogative of the 
President. 


National Treasury ensures 
that program performance 
information underpins 
planning, budgeting, 
implementation management 
and accountability reporting 
(D&S) 

Department of Public Service 
and Administration: 
increasing public service 
effectiveness and improving 
governance. Has established 
the Public Management 
Waich System (D&S) 


Auditor-General only has 
powers of reporting and 
recommendation. (D&S) 


Department of Cooperative 
Governance develops 
national policies and 
legislation (D&S) 


Office of the Premier 


National Integrated 
Monitoring and Evaluation 
System, led by Ministry of 
Planning and National 
Development, was 
established in 2008. (D&S) 


The Vision 2030 Board and 
the Vision Delivery 
Secretariat, a semi- 
autonomous government 
agency that also sits in the 
Ministry of Planning, shares 
monitoring and evaluation 
responsibility with MED. 
(D&S) 

Others entities involved in 
M&E: Ministry of Finance 
and National Treasury, the 
Auditor-General, and the 
Kenya National Bureau of 
Statistics. Others include 
ministries with sector- 
focused M&E efforts, such as 
health, agriculture, and 
livestock (CLEAR-Wits 2012, 
p. 86; Kenya School of 
Monetary Studies 2010, pp. 
39-40). (D&S) P. 22 


The National Development 
Planning Commission 
(NDPC), an independent 
government agency, was 
established in 1992 under the 
Constitution (§87 2 e) as the 
lead agency in monitoring, 
evaluating, and coordinating 
development policies. (D&S) 
PAS 


NDPC recognised that 
although they had done some 
assessment in government, it 
was not comprehensive. 
Consequently it would be 
difficult to address on-going 
issues, as there was no 
picture of who, what and 
where people were engaged 
with M&E in the country. 
Proposal, p. 6 


The Policy Evaluation and 
Oversight Unit was 
established within the Office of 
the Presidency in 2009. Is 
perceived to be a political 
instrument. (D&S) P. 16 
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develops Provincial Growth 
and Development Strategy 
that guides the planning and 
M&E function within each 
province (D&S) p. 39 
Statistic's South Africa 
Public Service Commission 
reports directly to Parliament, 
investigating, monitoring, and 
evaluating the organization 
and administration of the 
public service. (D&S) 

Line departments, 
provinces and 
municipalities are expected 
to set up their own M&E 
systems, aligned with the 
GWM&E Framework. (D&S) 
P.40 

Cabinet-level clusters: 
increased role in M&E in May 
2009. The appointment of a 
new President and 
administration. (D&S) 





Leaders/Champions Good support for M&E atthe Lack of champions Lack of champions The government of Uganda 
ministerial levels, thus has strong and well-placed 
helping its lead evaluation evaluation champions in the 
agency advance its agenda offices of the President and 
(D&S) p. 9 the Prime Minister (D&S) 

(p.9) 

Other M&E forum to ensure M&E Annual Public Expenditure Website Evidence-Based PM 
and planning functions Review to benchmark system established, and 
coordination at the provincial Kenya's economic accessible to government 
level in all provinces. (D&S) management against stakeholders (D&S) p.18 
P. 40 selected peer middle-income 


Departments conducted their Countries. (D&S) 
own evaluations and had no Public Expenditure Tracking 
systematic way to share Surveys are coordinated by 
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Resources 


findings. Their efforts 
remained focused on 
monitoring. 


The Department of 
Performance, Monitoring and 
Evaluation uses the 
Management Performance 
Assessment Tool to assess 
departments’ management 
performance. 


The Department of 
Performance, Monitoring and 
Evaluation: lts 2012 budget 
was $20.8 million and it had 
197 total staff. (D&S) p. 38 


the Monitoring and 
Evaluation Directorate and 
used to identify leakages in 
finances. (D&S) 


MED’s actual budget for 
2011 ($1.3 million) was a 
fraction of its planned budget 
($3.8 million), as a result of 
the political crisis of 2007-08 
and the ensuing economic 
setback. After covering other 
costs, MED had 
approximately $400,000 left 
to dedicate to M&E efforts 
(CLEAR-Wits 2012, p. 79). 
(D&S) P. 21 


Not enough staff for MED. 


In 2010, of a total M&E budget 
of approximately $11.1 million, 
less than three per cent was 
spent on evaluation. In 
contrast, 70 per cent was 
spent on monitoring (CLEAR- 
Wits 2012, pp. 61, 65). (D&S) 
p. 17 


Budgets for monitoring, 
evaluation, and related 
capacity-building efforts are 
described as insufficient 
(D&S) (p. 11) 


The vast majority dedicated to 
monitoring (and the majority is 
spent on capacity building and 
field visits, rather than data 
collection, management, 
analysis, reporting, and 
dissemination) (D&S) p. 11. 
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Researchers 
Evaluators 


South African Monitoring 
and Evaluation Association, 
created in 2005, benefits 
from government 
engagement, which 
includes government 
participation in and funding 
for the association 

Public Administration 
Leadership and 
Management Academy 


M&E learning network, 
established by the 
government. 


University of Stellenbosch, 
School of Public 
Management and Planning 


Council for Scientific and 
Industrial Research 


Human Sciences Research 
Council designed the 
Expanded Public Works 
Programme M&E 
framework 

An increasing number of 


consulting firms undertake 
M&E work (D&S) p. 41 


NGOs and private 
organizations are involved 
in government’s M&E 
efforts: Mars Group and 
the Institute of Economic 
Affairs, Kenya Association 
of Manufacturers, the 
Kenya Private Sector 
Alliance, and professional 
bodies such as the Law 
Society of Kenya and the 
Institute of Public 
Accountants of Kenya, the 
Kenya Institute of Public 
Policy Research and 
Analysis, Institute of Policy 
Analysis and Research, 
Center for Economic and 
Social Right — Kenya, 
Professionals in 
Measurements Network 
and the Evaluation Society 
of Kenya. (D&S) p.24 


External institutions 
investing in M&E: Danish 
International Development 
Agency, The World Bank’s 
Global HIV/AIDS Program 
hosts the Global HIV/AIDS 
Monitoring and Evaluation 
Team, The World Health 
Organization, The US 
Agency for International 
Development, The African 
Medical and Research 
Foundation (D&S) 


Focuses on training: 


Ghana Institute of 
Management and Public 
Administration. (1961) and 
The Institute of Statistical, 
Social and Economic 
Research. (D&S) P. 18 
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N.2 Latin America®? 





“Carrots” 


“Sticks” Mexico’s constitutional reform regarding Performance Based Budgeting (PBB) 
induced an unprecedented increase in demand for M&E at the sub-national level. 
(Proposal, p.1) 
Laws: General Law of Social Development, Federal Law of Budget and Fiscal 
Responsibility (Proposal Annex, p. 23) 


The Ministry of Finance and Public Credit (SHCP), for instance, keeps control of 
budget performance indicators. (Proposal Annex, p. 33) 


The two institutions that have fully developed M&E systems are the Ministry of 
Public Administration (SFP) and the National Council for the Evaluation of Social 
Development Policy (CONEVAL) (Proposal Annex, p. 33) 

The National Evaluation Council on Social Development Policy is responsible for 
monitoring and evaluating the coverage, management, services, results, quality and 
impact of social programs. 


“Sermons” The recommendations and conclusions of CONEVAL evaluations are not legally or 
administratively binding. 


Honduras is in the process of building 
institutions and M&E systems by creating a 
Social Development Ministry that is developing 
a Capacity Building Strategic Plan regarding 
public and social policy making, planning, 
monitoring, evaluation and systematization. 
(Proposal, p.2) 


Countries with developed systems or 
regulations: Argentina, Chile, Colombia, Costa 
Rica, México, Uruguay 

Countries with new/ incipient/ not fully functional 
M&E systems or regulations: Bolivia, Dominican 
Republic, Ecuador, El Salvador, Guatemala, 
Honduras, Panama, Paraguay, Peru 


No M/E systems or regulations: Nicaragua, 
Venezuela (Proposal Annex, p. 21) 


M&E entities Ministry of Finance and public Credit (SHCP) 
Institutions that have fully developed M&E systems: 
Ministry of Public Administration (SFP), Ministry of Social Development, 
National Council for the Evaluation of Social Development Policy (CONEVAL) 


The Unit for Public Management Evaluation and Performance has three 
components: 


Performance evaluation system (SED) 


39 Main source: CIDE Proposal to Host the CLEAR Centre in Spanish-speaking Latin America. 
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Annual Evaluation System (PAE) 
Integrated Information System of Government (SIIPP-G) 
Resources Mexico total Budget: $192,073,818,548 USD, M&E stipulated for 2011: 28,750,000 Percentage M&E of total budget (Proposal 

USD Percentage of total Budget: 0.01% (Proposal Annex, p. 22) Annex, p. 22): 
Argentina: 0.02% 
Chile: 1.3% 
Colombia: 0.1% 
Ecuador: 0% 
El Salvador: 0% 
Guatemala: 0% 
Peru: 0.01% 


Researchers Includes the M&E Latin-American Network (redlacme) 
Evaluators 





N.3 South Asia 


“Carrots” Donors support evaluations of programs that they financially support. USAID awarded a 5-year, $71 million task order to 
Want to move towards performance related incentive (pay) scheme for Management Systems International (contractor) to 
government departments implement the Independent Monitoring and Evaluation 
Program. 
“Sticks” Demand for greater accountability from Media, Civil society activism — The PRSP Secretariat housed in the Ministry of Finance has 
e.g. community score cards, citizen score card, social audits, been mandated with the overall lead in coordinating, 
Comptroller and Auditor General (CAG) and Supreme Court monitoring, evaluating and tracking the implementation of 


social indicators originally identified in the |-PRSP42 For this 
40 Various sources, including: World Bank, (2013):” The Government Monitoring and Evaluation System in India: A Work in Progress. ECD Working 
Paper No.28, and CLEAR centre documents, including J-PAL proposal and annual centre reports. 
41 Various sources, including J-PAL proposal and annual centre reports 


42 http:/Awww.finance.gov.pk/prsp_sectt.html 
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Category India* 


“Sermons” 


Programs of national importance (especially of the rural development 
and the social sector ministries) are also the flagship programs of the 
political party in power. 

2009: government introduced performance management system in 
union government — requires each department to prepare a results 
framework document (RFD). Not yet effectively implemented. 
Outcome budgeting introduced in 2005, but not yet effectively 
implemented. 

Right to information Act (2005) — contributes to overall trend towards 
more transparency, though mostly relevant for access of individuals to 
information 


Programs of national importance (especially of the rural development 
and the social sector ministries) are also the flagship programs of the 
political party in power 

International donors promote M&E 


Pakistan“! 


purpose a mechanism has been instituted with the Controller 
General of Accounts (CGA) for quarterly tracking of anti- 
poverty expenditures. 


Policies outlined in PRSP have been linked with the 
achievement of key social and human development goals. 
However, depending upon the variable being monitored, 
there is an implementation lag between expenditures 
incurred and outcomes achieved which makes it difficult to 
assess policy performance immediately. 


Report of the National Commission for Government Reforms 
on Reforming the Government in Pakistan (2012) notes a 
need for proper performance evaluation. This has not yet 
been implemented 


State governments do not prioritize M&E 
M&E are not always a top priority when designing and 


implementing policies. It is currently mainly donors who are 
promoting M&E. 


M&E entities 


Planning Commission’s Program Evaluation Organization (PEO) — 
historically charged with evaluating central government programs, 
created 1952. Had field offices, but mostly closed. Decline in 1970s, 
gradual improvements since 1990s. Remains anchor of evaluation 
work, but is under-staffed, and has limited influence on line 
ministries/state level 


2009: Creation of Development Monitoring Unit in PMO 


Performance Management and Evaluation System located in Cabinet 
Secretariat 

2013: Planning Commission established Independent Evaluation Office 
(IEO) — likely to absorb staff of PEO. Intended to exercise influence 
over line ministries. 

IEO budget generated from government of India, but independent from 
Planning Commission 


43 http://www.dgmepunjab.gov.pk/ 


PRSP Secretariat housed in the Ministry of Finance 


Director General M&E. Planning and Development 
Department Government of Punjab*8 


Provincial level planning and development department and a 
federal version that includes a planning Commission 
charged with undertaking research studies and state policy 
development initiatives. Reputation of these bodies mixed. 
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No legislation underpinning creation of IEO. 


Positive: IEO head is ex officio member of planning commission; holds 
rank of Minister of State, i.e. above line Ministers 


Concurrent Evaluation Office (Ministry of Rural Development) 





Resources Government recommended that 2 per cent of program funds be set 
aside for evaluation. But large part of funds controlled by line ministries, 
which have resisted critical evaluations. 


Researchers Often outsourced by PEO to research institutions. Often attached to a Availability of internal and external training limited. 


Evaluators particular ministry. E.g. National Institute of Rural Development Lack of structures and skills to undertake rigorous research 
conducts evaluations for Ministry of Rural Development. Ministry of and evaluation October 2013 — first meeting of Pakistani 
Human Resource Development contracts National University of community of evaluators 


Educational Planning and Administration and National Council of 
Educational Research and Training. 


Also independent institutions funded by Indian Council for Social 
Science Research. 


Not many credible research institutions funded by government. 


Most state programs are being evaluated, but institutional and human 
capacity to conduct evaluations focusing on outcomes is limited. 
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N.4 Francophone Africa“ 





“Carrots” Weak and misunderstood culture 
of evaluation. 


“Sticks” There is a weak domestic Donors are almost the only ones M&E units have been established in all 
demand for M&E, especially demanding M&E. Most projects ministries, but generally there are too 
for projects and programs with foreign funding have an M&E few staff, who are often not well trained 


not financed or driven by unit. P. 18 on M&E. 


donors or technical partners. 


On-going reforms since 
2001, though largely driven 


For government, M&E is not 
perceived as a priority. P.18 


In the few cases where M&E 


Civil society organizations (and the 
private sector through the Private 
Sector Cluster) involved in preparation 


by donors and international 
requirements. P. 21 

Economic Policy Monitoring 
and Coordination Unit leads 


of progress reports of the GPRS 


under-resourced and generally Civil society organizations also 
ignored. p18 produce alternate reports on poverty 
e Civil society involved in developing and MDGs via the Social Watch 

an M&E plan of the national and implementing the national Network. P. 30 

economic and social policy. plan, but questions of the 

P. 22 effectiveness of their involvement. 

No central policy or PAS 


standards for performance Focus currently is on control and 
monitoring or evaluation. P. fight against corruption, with an 
22 auditing approach. In this 

Some ministries have an environment, M&E and PM not 
M&E unit (for example, well understood or perceived. P.19 
Health, Education, 

Agriculture, Water, Justice), 

but most ministries have 

neither the skilled human 

resources nor sufficient 

budgets to carry out their 

mandate. 


mechanisms do exist, they are 


44 Main sources: An Assessment of Demand for Monitoring, Evaluation, and Performance Management Information and Services, CLEAR, 2013, and 
CESAG proposal to host CLEAR centre. 
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M&E is not well integrated 
into the organization. P. 23 


A number of disparate 
nongovernmental 
organization efforts at 
building evaluation culture 
among civil society (for 
example, Citizen Control of 
Public Action). P. 24 


Perceived need to raise 
awareness and 
understanding of potential 
uses of M&E at all levels, 
including political and senior 
government officials. P. 21 


“Sermons” 


Burkina Faso [Mauritania CI 


Monitoring and 
evaluation is 
elevated to a 
national priority 
and is the subject 
of attention. 


This country has 
expressed a clear 
interest in the 
implementation 
and development 
of monitoring and 
evaluation 
activities and of 
results-based 
management by 
developing 


2009 election brought 
“normalization” of the political 
situation and resumed cooperation 
from international partners. P. 18 


On-going reforms provide a 
favourable context for M&E 
development (3-year reform 
program signed with IMF, 2010- 
2012) p.18 


Perceived need for more 
advocacy and awareness of need 
to improve public expenditure 
accountability. P. 18 


national strategies 
(CESAG proposal, 
p. 2) 

Need for training in 
the culture of 
evaluation and 
capacity building in 
monitoring and 
evaluation and 


RBM for the 
benefit of staff in 


Indication by Director of Cabinet that 
M&E and PM are priorities of 
government: growth of evaluative 
practice over the past 5 years and 
creation of the Bureau for the 
Evaluation of Public Policy (BEPP). P. 
28 


More recent reforms (last two years), 
aimed at accountability and 


governance, have M&E implications p. 


28 

Much of the focus driving the M&E 
system is on progress reporting 
against the Growth and Poverty 
Reduction Strategy (GPRS) p. 30 
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charge of these 
matters in 
government 
(CESAG proposal, 


p. 2) 


M&E entities 


Other 


Resources 


National committees for 
M&E of public policies and 
programs; government- 
managed Projects and 
Programs Implementation 
Support Unit (PPISU); 
ministry-level planning and 
monitoring units. P. 21 


But: these still lack details 
around their mandate and 
composition. P. 24 


A number of organizations 
with some responsibilities for 
M&E, but not well 
harmonized or coordinated. 
P. 22 


National Statistics and 
Demography Agency. 
Though some sectors 
(Health, Education) have 
better data capture systems, 
generally there are concerns 
for lack of harmonization of 
data collection procedures 
and reliability of data. 


Few resources are dedicated 


Few M&E resources and skilled 


Central-level M&E bodies: BEPP, 
Economic and Financial Programs 
Monitoring Unit (EFPMU) and 
Directorate General for Monitoring of 
Projects and Programs (DGMPP); 
BEPP within the Prime Minister’s 
Office plays the coordinating role for 
M&E. p. 27 


Bureau for the Evaluation of Public 
Policy (BEPP) in the Performance 
Management Office is the central M&E 
Unit for the M&E system for public 
policies. P. 31 


GPRS M&E system; a new NPDA 
M&E system, not yet implemented; 
and the M&E system for public 
policies. P. 28 


Annual funding allocated to M&E 
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to structures in charge of M&E specialists or evaluators. P. 
M&E 19 
Supply of M&E A E 
Researchers Consultative Committee of Mauritanian Association for M&E technical group of international 
Evaluators Technical and Financial Monitoring and Evaluation and partners and donors, currently 
Partners: Coordinating donor Mauritania Perspectives p. 20 coordinated by UNDP 
group. P. 24 Benin Evaluation Association and 
National Evaluation National Network for Impact Analysis 
Association promotes p. 30 


evaluation culture through 
training and development 
initiatives p. 24 
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Annex O Types of Results Achieved 


Strengthened capacity to produce & use evidence 


Increased awareness of: 


Increased awareness of: 


What M&E are, and 
what their benefits 
can be in the given 
context 

Specific types of 
evaluation 
approaches and 
related 
methodologies 


Existing gaps/needs 
in individuals’ 
knowledge or skills 
Existing gaps in 
organisational 
capacity for M&E 
and/or RBM 


New or improved 
knowledge and/or skills 


Related to specific 


All centres: Consulted clients acknowledged that capacity building events provided by the respective CLEAR centre had 
provided them with relevant examples of how evaluations have been conducted and/or used in similar contexts, often 
examples taken from their own country or region. 


All centres: Post training assessments conducted by the centres, as well as consultation with participants of CLEAR 
capacity building services (in particular training) showed that clients had gained new insights/knowledge on various types 
of evaluation, in particular Impact Evaluation. Several consulted individuals noted that they already had, or were planning 
to implement impact evaluations in their own organisation (to varying degrees of complexity). Others acknowledged that 
even if they did not expect to be able to implement a full-fledged impact evaluation due to financial and/or staff 
constraints, they appreciated the insights gained into when, why and how IE can be applied. Some stakeholders noted 
that this helped them better understand and position the benefits and limitations of evaluation approaches currently used 
by them in a broader context. 


East Asia: Consultations with CLEAR clients indicated that Centre training has changed government officials’ perception 
on management, shifting their focus from inputs to outcomes (results). Clients also noted that the training had 
contributed to changing their perception on the nature and purpose of monitoring and evaluation, putting less focus on 
control and criticism or punishment, and stronger emphasis on learning. 


South Asia: One consulted CLEAR client in Pakistan who had attended a 5-day training course on M&E offered by 
CLEAR noted that the training had significantly changed his views on what constituted ‘good’, i.e. solid and rigorous data 
collection. He had not heard of randomised control trials beforehand, and felt that related insights were very useful to him 
and also to his whole team. 


East Asia: Consulted Chinese government officials acknowledged that CLEAR training had helped them conduct 
research into existing gaps between international good practice and domestic practice on M&E at the time. This 
realisation contributed to the MoF issuing new Regulations on Performance Evaluation on IFI projects that are informed 
by international practice. 

Latin America: The LA centre conducted meta evaluations for various social programmes under the Social 
Development Ministry in Mexico. These are expected to help inform the ministry’s future programmes. Another example: 
The government of Argentina requested technical assistance and training from the LA centre following an international 
seminar on M&E organised by the CLEAR centre. 

South Asia: Clients requesting more in depth services after having attended a more generic capacity building event 
were reported. 


Anglophone Africa: Interviewed PDME staff in RSA claim that their knowledge and skills have increased in the areas 
covered by in-service training supported by CLEAR AA. 


“| have taken the lead in designing impact evaluation studies at the school. Currently, | am implementing an impact 
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Types of Benefits 


evaluation or RBM 
tools 


e For planning, 
managing/conductin 
g evaluations, 
including specific 
types of evaluation 
e planning and 
conducting M&E 
capacity building 
events 
Increased confidence 
and/or motivation to 
explore and/or address 
specific M&E issues in 
more depth 


Access to larger number 
of staff with 
knowledge/awareness 
of M&E 


Changes in 
organisational culture (in 
‘how things are done’) 


Changes in planning for, 
conduct of, and/or use of 


assessment for Kenya School of Government's (KSG) flagship programme; the strategic leadership development 
programme (SLDP). (2) In 2013, | organized a policy research workshop for the school's faculty in which part of the 
training was on technical approaches to impact evaluations. (3). am currently a lead trainer in impact evaluations.” 
Participant in the AA IE Course 


Latin America: CLEAR LA centre gave a course in Argentina on Logic Model. Survey results show that participants 
acquired/increased their knowledge on the use of the Logic Model Matrix for their work (e.g. for better project planning 
and implementation, for project evaluation). Another course given by CLEAR LA (in Mexico, involving government staff 
also from Costa Rica, Peru, Argentina, Nicaragua, Uruguay, Colombia, and Venezuela) was on Performance-Based 
Budgeting. Survey results show that participants acquired/increased their knowledge on how to use PBB in the budget 
process and how to better link the planning phase with the budgeting phase. There is no information yet on whether and 
how participants have applied these knowledge and skills. 


Anglophone Africa: Consulted participants of the CLEAR Impact Evaluation course noted that: 
“Now | do evaluation with confidence and always ensure that the counter-factual is taken care of.” 
“| got more confidence on designing evaluations by collecting data before a program start.” 


Latin America: State governments in Mexico have started contacting CLEAR LA to get technical assistance on M&E at 
the sub-national level, indicating interest (motivation) to improve their related work 


South Asia: In India, the CLEAR centre is working with a number of state level governments to assess and help improve 
their respective monitoring systems. 


South Asia: One consulted client who had participated in training on M&E said that since returning from the course he 
has started to actively look online for additional information and tools on the use of RCTs in particular, and on evaluation 
in general. 


East Asia: Almost all the MoF staff working in the divisions dealing with international aid have by now received 
SHIPDET training. Now the training is being provided to other MoF staff and public servants in other departments. 


South Asia: In 2013, CLEAR SA organised training on M&E for Indian Economic Services (IES) Probation Officers, and 
as (in a separate event) as part of the in-service training for mid-level officers. The workshops focused on the use of 
impact evaluation to inform policymaking and programme design, and addressed survey design and data collection for 
impact evaluation. The Indian government has expressed its interest in making the courses a regular part of its 
curriculum for preparatory and in-service training. CLEAR SA hopes that by addressing probation officers who are at the 
beginning of their career, the training will contribute to increased M&E awareness and basic skills across the IES staff. 


Anglophone Africa: CLEAR is one of several RSA institutions helping DPME deliver training local/provincial 
government staff all over RSA in M&E. 


South Asia: CLEAR SA worked with the NGO Breakthrough to institutionalise and improve its M&E processes, in 
particular its ability to plan for and manage an impact evaluation of one of its projects using RCT. Also, the centre has 
provided technical assistance on evaluation to the Government of Haryana Education Department and its Centre for 
Research and Experiments for Action and Policy (REAP) to oversee the M&E of on-going projects such as a Midday 
Meal Scheme, teacher training, and Meena Manch (Adolescent Girl Clubs) to inform education policies. 
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Types of Benefits 


monitoring and/or 
evaluation 


Better quality of 
evaluations produced 


Availability of 
organization specific 
toolkits, guidelines, 
policies, etc for 
monitoring and/or 
evaluation 


Anglophone Africa: Participants AA IE course commented: 


“Since the training, my organisation has been more aware of multiple tools for impact assessment and we have used 
several tools for impact analysis and sensitivity assessment.” 


“Our organization reconsidered my advice based on the training to change the methodology for the upcoming impact 
assessment for our projects.” 


“We have extensively changed the approaches of the consultancies that we do in measuring impact where we have put 
more focus on establishing the counter factual alongside the usual before/after for concrete evidence in suggesting 
attribution of impact of projects and programs.” 


Francophone Africa: In Senegal, the President's Cabinet office has approached CLEAR centre to support the 
president’s commitment to rolling out a results-focused culture throughout the government. 


Anglophone Africa: “More robust evaluation reports produced by M&E section as a consequence of the training on 
Impact Evaluation” AA IE course participant. 


South Asia: CLEAR SA won a competitive bid issued by the local USAID office to help the agency integrate impact 
evaluation for learning and accountability. USAID has a global evaluation policy that includes impact evaluation, but the 
India mission has never commissioned one before. The centre developed an impact evaluation toolkit, and worked with 
various technical teams within the agency to strengthen their understanding and skills related to planning for and 
managing impact evaluations of USAID-supported initiatives. 


Anglophone Africa: CLEAR AA has provided various types of support to the DPME in RSA which has included working 
alongside it in the development and delivery of five in-service training courses that covered topics including how to 
manage evaluations, how to prepare management responses to evaluation, how to communicate evaluation results, how 
to develop TOC and logframes. CLEAR also assisted DPME in developing “Evaluation design clinics” for staff. 


Expanded professional expertise in regions 


More and/or better 
opportunities (in terms of 
offered content) for 
networking among 
relevant stakeholders in 
the region and/or globally 


All regions: Centres sponsored clients/partners to attend CLEAR regional forum in Mexico. 


South Asia: The centre sponsored the South Asian Evaluation conclave 2013 in Nepal and contributed to regional 
consultations and task committees; led plenary session and panel discussions on evaluation capacity building and 
leveraging data and technology for impact evaluations, and panel discussion on evidence based decision making, and 
two-day workshop on impact evaluation theory and practice; helped foster ties and deepen collaboration with different 
actors, such as SLEVA. 


Anglophone Africa: The South African DPME appreciated the connections and exchanges that they have had with the 
aid of CLEAR support for a study tour to CONEVAL, the Centre of Excellence in Evaluation in Canada, and to other 
countries (Colombia, Mexico, US) that are also striving to develop government evaluation capacities and performance. In 
addition, DPME itself plays an active role in reaching out to others in the South to share experiences. CLEAR AA has 
been able to bring in internationally known experts to work with national stakeholders in RSA, Kenya, and South Sudan. 
Also, CLEAR AA staff has participated in Afrea annual conferences, in particular on work related to “Made in Africa” 
deliberations., The team is also involved in SAMEA where it is seen to help build on best practice and international 
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New/improved training 
curricula and/or 
programmes on M&E in 
one or more academic 
institutions 


thinking in evaluation. 


Latin America: In the CIDE seminar on public safety, to which CLEAR-LA contributed with an M&E module, it was 
possible for the first time to bring together government representatives, CSOs, experts in public safety, and evaluators. 
CLEAR LA supported exchanges with the centres in Anglophone Africa and South Asia. 


Anglophone Africa: The AA centre contributed to the creation of a diploma and a Master’s programme in M&E at Wits 
University. The diploma programme had 40 students in its first cohort in 2013, selected from 400 applicants. It is 
expected to have 50 in each subsequent year. 


Latin America: The LA centre contributed to the development of a graduate programme in M&E, consisting of modules 
in various universities’ existing postgraduate programmes (e.g. in public administration, migration, public safety), as well 
as to the development and delivery of courses and workshops given during conferences in the region organised by 
CLAD, REDLACME, RELAC. To date, a total of 447 individuals from 16 countries in the LAC region have attended. 


Improved enabling environment & demand for M&E 


New/improved knowledge 
on existing demand for, 
and/or existing structures 
and provisions for RBM 
and/or M&E in one or 
more countries. 


Facilitating discussion 
and reflection on the role 
of M&E in a variety of 
sectors/areas 


Anglophone Africa: Exploratory case studies on African Monitoring and Evaluation Systems led by CLEAR AA and 
DPME using Benin, Burundi, Ghana, Kenya, Senegal, South Africa and Uganda as cases have become reference 
material for understanding the (Southern) African context for M&E. Several consulted stakeholders saw the potential of 
the publication to help governments develop or strengthen M&E system on the continent. 


Francophone Africa: Assessment of demand in M&E in Benin, Mauritania and Senegal first study of its kind in the 
region. Consulted stakeholders acknowledged the study had raised interest but also expectations of follow-up among 
various actors in the region, which have not yet been addressed. 


East Asia: CLEAR training contributed to Chinese government officials realising a gap between international practice 
and domestic practice on M&E. As a result, they issued the Regulations on Performance Evaluation on IFI Projects 
based on international practice. 


South Asia: Series of 10 roundtable discussions to date on M&E topics conducted in various locations in India in 
collaboration with other organisations, including for example UN Women on the roundtable on gender and evaluation. 
The events have been attended by 199 people to date, and by 900 viewers online. Consulted stakeholders in India 
expressed their appreciation for the events, which they saw as unique opportunities for knowledge sharing, awareness 
building and networking among diverse actors involved in evaluation. 


Similarly, in Pakistan CLEAR SA facilitated a Policy Dialogue on Education Support Services, putting special emphasis 
on backing interventions by data-driven evidence and building effective M&E systems to ensure that the desire impact is 
achieved. 

Latin America: CLEAR-LA supported the expansion of M&E to other areas than social policies/programmes (e.g. public 
safety). In a CIDE seminar on public safety, to which CLEAR-LA contributed with an M&E module, it was possible for the 
first time to bring together government representatives, CSOs, experts in public safety, and evaluators. 


Anglophone Africa: The AA centre co-organised South Round Table Discussions in Pretoria (South Africa) to 
implement a South-South dialogue on government M&E systems. 
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Types of Benefits 


New policies, 
frameworks, guidelines or 
other factors likely to 
create or strengthen 
existing incentives for 
M&E/RBM at national or 
sub-national levels (i.e. 
within M&E systems) 


Changes in how 
government units 
approach/use M&E 


Innovations in M&E 


“| have sold the concept of Impact Evaluation in our social protection (SP) sector (sector wide approach) and discussed 
with some officials at the Prime Minister's office about the benefits of IE. DfID intends to sponsor an IE of one its major 
programs in our country. ” AA IE course participant. 


East Asia: As a result of regional centre’s M&E capacity building, the city of Shanghai invited an expert from the centre 
to advise them on designing a municipal M&E system. Also, the Chinese MoF invited the centre to organise a seminar 
on best practices in M&E for provincial officials from financial bureaus to raise awareness on evaluation to support 
government plan to roll out larger government-wide M&E plan. Also, in March 2013, the MOF issued a Regulation on 
Performance Evaluation (PE) in of IFI loan projects in China. 


Latin America: LA centre supported the Peruvian ministry of women and vulnerable populations to create a monitoring 
index based on administrative and census information. The index will be used to assess how the ministry provides 
programmes and tracks results for vulnerable populations. 


Anglophone Africa: In Kenya, a two-week CLEAR training contributed to the Government of Kenya initiating further 
collaboration with CLEAR to roll out its M&E system in its 47 counties. In South Africa, CLEAR conducted a review of 
four DPME guidelines on different types of evaluations, which were included in the national evaluation plan. 


South Asia: In Pakistan, a CLEAR training participant who was amid-level government officer at the state-level (i.e. sub- 
national) responsible for M&E across sectors noted that his department had traditionally focused on monitoring only. He 
was the first in his team to attend a workshop on M&E at the national level (organised by CLEAR). The training 
significantly changed his perception and understanding of the potential benefits of evaluation, and he has since been 
repeatedly asked by his supervisors to present key insights derived from the training to others in the department. In his 
view, the training a changed not only his personal approach to M&E, but is making an impact in his department, where 
he observes an increased interest (including in his supervisor) to attend similar training, and to focus more on evaluation, 
as well as on more rigorous and systematic data collection through monitoring. For example, his department is now 
putting increased pressure on line ministries to submit project completion reports (which are required, but have 
traditionally not been submitted regularly), and is planning to expand that to enforce compliance for ministries to also 
conduct project evaluations and submit related reports. 


More/new discussion and 
exchange among diverse 
stakeholders. 


More/first time 
explorations of the use of 
M&E in specific sectors 


South Asia: Series of roundtables (see above) 

Latin America: In the CIDE seminar on public safety, itwas possible for the first time to bring together government 
representatives, CSOs, experts in public safety, and evaluators. 

Anglophone Africa: CLEAR collaborated with Afrea (with Rockefeller support) on a regional seminar to explore a Made 
in Africa approach to evaluation. Consulted stakeholders also noted, however, that there is no evidence yet whether this 
event will have further effects at regional or national levels, some see it as a lost opportunity for change. 


South Asia: In Pakistan a participant of an M&E training workshop conducted by CLEAR noted that as part of an 
exercise during the training he and other participants developed the idea to use mobile technology to track school 
absentees. Upon his return to his government office, he presented this idea to his supervisor and was met with 
enthusiastic response. He and his team are now working to implement this idea. 
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Types of Benefits 


Publications 


Latin America: Broadening the reach of M&E to: i) policies other than social development policies (e.g. the evaluation of 
the implementation of conditional cash transfers at times of natural disasters (publication under development); ii) other 
sectors than social policies (e.g. expansion of M&E training to areas traditionally excluded, such as public safety. 


South Asia — Mobile-Based Technology for Monitoring and Evaluation (2013). Consulted centre staff noted that the 
presented techniques were not ‘new’ in India, but probably for actors in many other parts of the world. Also materials 
accompanying/ summarising round table discussions and webinars on various M&E-related topics. 

Latin America: From Recommendations to Actions - Federal Programs Committed to the Evaluation Process (2011), as 
well as four other publications not published on the CLEAR website. 


Anglophone and Francophone Africa: Demand studies (see above) 
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Annex P Survey of Participants in IE Courses in Anglophone 


Africa 


P.1 Overview 


An invitation to participate in a survey on IE courses was sent to 95 potential respondents and 24 
completed surveys were returned (60 respondents did not open the invitation, 8 email addresses 
were not valid, and 3 viewed the survey but did not complete it). The survey and results are shown 


in section A.2 below. 


P.2 Survey questions and results 
Please identify which IE course you have taken: 
Response Chart Percentage Count 


Impact Evaluation Technical South Africa 
(July 2013) 


Impact Evaluation Policy South Africa 
(July 2013) 


Impact Evaluation Uganda (24 June - 6 
July, 2013) 





Total Responses 
Note: Two respondents selected both courses in South Africa, which is why the total is 23 rather than 25. 


Please identify your current employer category: 

Response Chart Percentage Count 
Government 

Civil Society 

Donor/ Multilateral Agency 


Other 





Total Responses 24 
Note: The 7 respondents who answered “other” in the question above provided the following information on their 
employer category: Research organisation (1); Kenya School of Government (1); consulting firm (1); academia (1); 
university (3). 
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Please state your opinion on the following aspects of the IE training that you participated 
in: 


Do not 
Somewhat Notatall know/ Total 
Relevant 
relevant relevant No Responses 
opinion 





Relevance of the training given your 
work responsibilities at the time of 
training 


18 
(75.0%) (25.0%) 0 (0.0%) 0(0.0%) 24 
Relevance of the training given your 21 


current work responsibilities ers eee D007) 00.0%) -24 


Relevance of the training given your 
employers' policies, priorities and/or 
resource allocations for evaluation 


14 


(58.3%) 8(883%) 1(42%) 1(42%) 24 


To what extent have you been able to utilize any of the acquired knowledge and/or skills 
you obtained from the training you received after you returned to your job? 





Response Chart Percentage Count 
Extensively 

Somewhat 

Not at all 4.2% 1 

Do not know/ No opinion 0.0% 0 


Total Responses 24 


Have you shared what you learned in the training with your colleagues? 


Response Chart Percentage Count 





Total Responses 23 
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Have you shared what you learned in the training with your supervisors? 


Response Chart Percentage Count 





Total Responses 24 


Please describe if/how any follow up any actions that you have taken since the training 
have contributed to any changes in your organization if any: 


1. Since the training, my organisation has been more aware of multiple tools for impact assessment 
and we have used several tools for impact analysis and sensitivity assessment. 


Am waiting to backstop consultants in the upcoming impact evaluation at the end of this year. 


Got more confidence on designing evaluations by collecting data before a program start. More 
robust evaluation reports produced by M&E Theme. 


4. | wrote proposal and still looking for funding 


We are doing control experiments than ever before in my organization. Unfortunately, we are a 
charity organization and we cannot afford the amount of resources we need for an impact 
evaluation. | have however sold the concept of IE in our social protection (SP) sector (sector wide 
approach) and discussed with some officials at the Prime Minister's office about the benefits of IE. 
DfID which co-chairs our SP sector intends to sponsor an IE of one it's major programs in 
Rwanda. 


6. (1). | have taken the lead in designing impact evaluation studies at the school. Currently, | am 
implementing an impact assessment for Kenya School of Government's (KSG) flagship 
programme; the strategic leadership development programme (SLDP). (2) In 2013, | organized a 
policy research workshop for the school's faculty in which part of the training was on technical 
approaches to impact evaluations. (3). | am currently a lead trainer in impact evaluations. 


7. | managed to get more interest in the work of 3ie, as such have extensively changed the 
approaches of the consultancies we do in measuring impact where we have put more focus on 
establishing the counter factual alongside the usual before after for concrete evidence in 
suggesting attribution of impact of projects and programs 


8. Our organization reconsidered my advice based on the training to change the methodology for 
the upcoming impact assessment for our projects 


9. impact evaluation technical has help me in evaluating the impact of projects and programs that 
the ministry of education is involved in. 


10. Now I do evaluation with confidence and always ensure that the counter-factual is taken care 
off... 


11. The organization adopted the impact evaluation methodologies and research. We are able to 
appreciate the use of evidence from impact evaluations. 


12. Minimal changes, by writing evaluation proposals 


13. 1. Improved training materials for my students 2. Published a paper on MeasureDHS website 3. 
Published a paper with BMC Medical Health 


14. Incorporated IE techniques in my research 


15. Clear understanding of what impact evaluation is and different approaches to evaluations has 
helped to guide our evaluation agenda. 
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16. Have moved to take further steps from monitoring to evaluation in my work. | am able to look at 
outcomes of projects other that what | used to do before. 


17. The organisation has now made impact evaluation a core input in the research program. 


18. Iam in the process of coming up with a Monitoring & Evaluation System for the Institute. | also 
taught a session on Complex Impact Evaluations to our Post Graduate Diploma in M&E students. 


Do you have any other comments that you would like to make at this time about the Impact 
Evaluation course(s) you participated in (e.g., its quality, strengths, weaknesses etc): 


2. if possible......volunteer/internship attachments to Impact Evaluation Group centres to get hands 
on experience of what was learnt to willing participants. for a given period of time. 


3. lam happy with the content of the course and it is very relevant for organizations working trying to 
adopt result based M&E. The limitations are the trainers are not at equal level of competence. 
Some have relatively good experience while others are not good enough. The other point is that 
the course announcement did not reach the proper audiences. Some of the trainees have enough 
experience in IE. But you have to try to reach countries and organizations that deadly need the 
course. 


4. First of all, to say IE is very relevant for anyone involved in programs as a tool you need to be 
able to understand whether or not you making any impact by your interventions. The training in 
Uganda was however more of academic than professionally oriented for practitioners. Some 
resource persons were not up to the task. You would imagine, we were enrolled for a Ph D 
program. We did not have ample time to practically design IE models. Overall though, it was 
excellent, because for me it opened a whole new world of opportunities in the field and | 
recommend follow up trainings. 


5. Strengths: The programme imparts critical and useful technical knowledge on IE. Weakness: At 
10 days, the trainings are usually too short relative to the content to be delivered. Quality: | would 
rate the quality of the IE course at 3.7 out of 5. The amount of time for practical learning should 
be improved. Other Comments: It may help to convert the programme into a 2 year graduate 
programme complete with a practical IE project implemented by the candidates before graduating 
with an MSC. Hosting: The programme can be hosted by partner universities with the technical 
faculty capacity to deliver theoretical and practical IE training. 


6. The courses are relevant in order to advise impact based programming. In most of the teams | 
have engaged who are actually evaluation consultants only report distorted statistics such as the 
before after, not because it’s the easiest but because they have not had exposure to technical 
trainings as this. This training however requires a good statistical or econometric background. 
There is however need to also talk about participatory approaches. Little trainings are talked of 
especially in Africa. Projects such as USAID come with pre-planned M&E and at the end it does 
not yield the impact. Even if it is done, it may not give the diagnosis which if a blend of 
participatory approaches is used would advise the programming. 


7. it was so intensive yet run for a very short time. This affected the sharing of experiences among 
facilitators and participants. 


8. | think the users and beneficiaries of this course must be supported by the sponsors at all times 
for them to benefit from the course. 


9. The quality of the training was really good and the use of group team work and extra hours 
helped me as individual to understand some of the areas that seemed difficult during class hours. 
However, the time frame allocated for the course was not enough for me. | suggest that the 
course be allocated 3 - 4 weeks. 


10. i think it was rigorous and the quality was good 
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11. The training was of high quality but the content needed more time during the training. 

12. much was covered in a limited time. there is need to lengthen period for training 

13. Strength: Was very relevant to academician like me 

14. Increase the practical component of the course. May be case studies as group project. 

15. The quality was good but the course duration was inadequate 

16. Duration of the course may be too short to capture everything, therefore | suggested the course 
duration to be expanded to include some sort of home study maybe with some online guidance 
and then close up with the two weeks face to face training. The topics on statistics can be very 
tricky to those without such background and perhaps a basic module will help. Otherwise, impact 
evaluation is a very interesting course. 

17. The training should be extended to other beneficiaries because it very relevant for development 
practitioners. 

18. It was an excellent course. A great eye opener to what true IE looks like. 

19. None 

20. Its quality is excellent; however, it requires more than 2 weeks to avoid being over packed in a 
short time. 

21. | want to focus on the qualities and strengths of the training - it is very practical and timely 
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Annex Q IEG Functions of Governance and OECD Principles 
of Corporate Governance 


Sourcebook for Evaluating Global and Regional Partnership Programs 


Indicative Principles and Standards“ 


Functions of Governance 


12.4 The governing bodies of GRPPs typically exercise six core functions: 


Strategic direction. Exercising effective leadership that optimizes the use of the financial, 
human, social, and technological resources of the program. Establishing a vision or a 
mission for the program, reviewing and approving strategic documents, and establishing 
operational policies and guidelines. Continually monitoring the effectiveness of the 
program’s governance arrangements and making changes as needed. 


Management oversight. Monitoring managerial performance and program implementation, 
appointing key personnel, approving annual budgets and business plans, and overseeing 
major capital expenditures. Promoting high performance and efficient processes by 
establishing an appropriate balance between control by the governing body and 
entrepreneurship by the management unit. Monitoring compliance with all applicable laws 
and regulations, and with the regulations and procedures of the host organization, as the 
case may be. 


Stakeholder participation. Establishing policies for inclusion of stakeholders in 
programmatic activities. Ensuring adequate consultation, communication, transparency, 
and disclosure in relation to program stakeholders that are not represented on the 
governing bodies of the program. 


Risk management. Establishing a policy for managing risks and monitoring the 
implementation of the policy. Ensuring these core functions, and the criteria for assessing 
the performance of governing bodies in the standards section below, are adapted from the 
OECD Principles of Corporate Governance (2004). Although there exist other similar 
statements of such principles at the national level, the OECD Principles are the only set of 
corporate governance principles on which there is clear international consensus. Many 
governance functions for the for-profit private sector, as laid out in the OECD Principles, 
translate directly into equivalent functions for GRPPs (as well as for other public sector 
organizations, NGOs, and foundations). 


Conflict management. Monitoring and managing the potential conflicts of interest of 
members of the governing body and staff of the management unit. Monitoring and 
managing conflicting interests among program partners and participants, especially those 
that arise during the process of program implementation. 


Audit and evaluation. Ensuring the integrity of the program’s accounting and financial 
reporting systems, including independent audits. Setting evaluation policy, commissioning 
evaluations in a timely manner. 


45 Sourcebook for Evaluating Global and Regional Partnership Programs, Indicative Principles and 
Standards (2007). IEG-World Bank. http://www.worldbank.org/ieg/grpp Washington, D. 
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OECD Principles of Corporate Governance (2004) 


(Organisation for Economic Co-Operation and Development) 


The board should fulfil certain key functions, including: 


di 


8. 


Reviewing and guiding corporate strategy, major plans of action, risk policy, annual 
budgets and business plans; setting performance objectives; monitoring implementation 
and corporate performance; and overseeing major capital expenditures, acquisitions and 
divestitures. 


Monitoring the effectiveness of the company’s governance practices and making changes 
as needed. 


Selecting, compensating, monitoring and, when necessary, replacing key executives and 
overseeing succession planning. 


Aligning key executive and board remuneration with the longer term interests of the 
company and its shareholders. 


Ensuring a formal and transparent board nomination and election process. 


Monitoring and managing potential conflicts of interest of management, board members 
and shareholders, including misuse of corporate assets and abuse in related party 
transactions. 


Ensuring the integrity of the corporation’s accounting and financial reporting systems, 
including the independent audit, and that appropriate systems of control are in place, in 
particular, systems for risk management, financial and operational control, and compliance 
with the law and relevant standards. 


Overseeing the process of disclosure and communications. 


The board should be able to exercise objective independent judgement on corporate affairs. 


le 


3. 


Boards should consider assigning a sufficient number of non-executive board members 
capable of exercising independent judgement to tasks where there is a potential for conflict 
of interest. Examples of such key responsibilities are ensuring the integrity of financial and 
non-financial reporting, the review of related party transactions, nomination of board 
members and key executives, and board remuneration. 


When committees of the board are established, their mandate, composition and working 
procedures should be well defined and disclosed by the board. 


Board members should be able to commit themselves effectively to their responsibilities. 


In order to fulfil their responsibilities, board members should have access to accurate, relevant and 
timely information. 





e-Pact 


132 


CLEAR Midterm Evaluation — Annexes 





AnnexR Strategic Guidance Provided by the Secretariat to 
the Regional Centres 


Regional Centre Strategic/ Technical Advice Provided by the CLEAR Secretariat 


Anglophone Africa 


Francophone Africa 


South Asia 


Latin America 


East Asia 


Technical guidance and international support to kick start the centre’s 
research/knowledge work on country M&E systems (resulting in a number of 
studies that were used to assess how capacity building support could be 
provided). 


Advice for structuring the relationship and identifying challenges and areas for 
support to the Department of Performance Monitoring and Evaluation (DPME) 
of the South African Presidency, National Treasury, Parliament, AFREA and 
SAMEA (including collaboration for site visits and trips, contributions to 
conferences, structuring work programs and contracts, etc.). 


Identified issues with the implementation of the work program and 
management of the centre, alerted the Board, and initiated an Institutional 
assessment. Management of institutional assessment of the Centre to 
identify implementation and strategy issues and solutions. Led the finalization 
of the assessment and discussions with Wits. Provided support to the Centre 
for the design of the action plan emerging from the Institutional Assessment 
and continue to provide support and advice for its implementation. 


Support for the transition in Centre’s Directors (introductions to clients and 
stakeholders in South Africa and the region, support in strategic direction for 
the Centre) 


Guidance and peer review many of Centre’s materials (i.e. DFID studies, 
Darfur Studies, peer review of DPME M&E Guidelines, DPME Refection 
Workshops, etc.) 


Advice to the centre in the implementation of activities (Impact Evaluation 
Course in Uganda; support to participants in the IE course to develop 
proposals to the Strategic Impact Evaluation Fund — to enable them to learn 
by doing) 


Design and review of the Centre’s first major study — demand assessment 
across three countries, resulting in recognition of the centre and improved 
visibility at conferences and network events (working closely with and advising 
the international expert who worked with the centre). 


Feedback on work program, particularly on the new sub-national level work 
that is being conducted and the grass-roots work that will be initiated with 
ASER. Connected the centre to the HP Foundation and R4D, which is 
conducting an evaluation of ASER (still on-going and not public) to ensure that 
early findings are taken into account in formulating the approach 

Feedback on and inputs into strategy and context of M&E in the region — 
providing research/background information that the centre did not have good 
access to previously 


Provided peer review comments for projects such as the Evaluation Outlook, 
Utilization of evaluations, etc. 


Advised the Centre in the drafting of the Agenda for the Peer-Learning section 
of the Global Forum, which focused on the experiences in M&E of the Latin 
America region. 


Leveraged additional funds for the centre through the IDB 


Helped establish the centre’s profile by including it in the high-level 
international seminar on evaluation requested by the Ministry of Finance 
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Regional Centre Strategic/ Technical Advice Provided by the CLEAR Secretariat 


Global — Across All 
Centres 


(which was requested from the WB) 


Templates for chunky, project type work so that objectives become clear, 
rationale is well grounded and there is a system for M&E 


Modalities for capacity building in the ToC have pushed/encouraged them to 
think beyond training. See http://www.theclearinitiative.org/ECD-CLEAR- 
ChangeAgents-ICOs-CD-Activities 4October%202013.pdf 


Quality assurance guidelines (templates for participant feedback, feedback on 
advisory services/technical assistance, project development, tracer studies, 
and so forth) 


Suggestions on RAC members 


[The items below are typically of team leaders of Bank projects — to keep 
abreast of issues/provide advice/knowledge, deploy technical assistance as 
needed, and so forth.] 


Technical review of project document — plus getting it internally reviewed. The 
project document must outline the context, strategy, and specific 
implementation plan. 

On-going review of and advice on identifying and addressing issues related to 
overall strategy, work program, procurement, disbursement 

“Scanning” of development environment and keeping the centres informed of 
those issues (from the Bank, evaluation associations, etc.) 

Continued advice on business planning — working with an expert and thinking 
through different models individually with the centres (e.g., think tank model, 
“affiliate model” and so forth) 

Advice on who to include for global fora, identifying strategic clients, and so 
forth 

Obtaining country clearances for the work to proceed in-country through the 
grants — coordinating with Ministries of Finance/Treasury, WB offices, etc. 
Providing technical assistance on specific services, products, etc. (more with 
the African Centres) 





e-Pact 


134 


CLEAR MTE 


Factual Corrections, Key Comments, and Clarifications 


From Secretariat 


Factual corrections noted in red 


| Page No. | Issue = sid Correction/key comment/clarification Response to key comment! 
Secretariat comments 


i CLEAR is financially 
supported by 12 partners 


ii No definition of what 
“success” looks like in 
development terms 


iii Measuring “success” in terms 
of development results 


ii Internal and external contexts 
varies significantly and were 
not taken into account in 
program design 


11 partners — not 12 Factual error corrected in final report 


CLEAR has a results framework and specific indicators 
in development terms. Another conclusion that the 
evaluation may wish to consider is that “success” was 
defined at too high a level for a program of this nature; 
defining success in “development terms” may be 
inappropriate for a program this size. The “global” 
success was to have established centers addressing 
evaluation capacity and the medium-term outcomes the 
centers achieve in the context of those strategies, 
working with relevant stakeholders in ways that make 
sense to move the specific national and regional M&E 
agendas forward. 


It is unclear what the evaluation means by “measuring 
success in terms of development results” — an example 
would be useful 


The program commissioned regional demand studies, 
regional consultations, and competitive bids. These 
contained information on contexts that informed program 
design. It would be helpful to understand what else 
should have been done. 


t Comments for which no response is noted in this column have been noted with thanks but have not led the evaluation team to make changes to the report. 


= 


| Page No. | Issue n Correction/key comment/clarification Response to key comment! 


5, and 
throughout 
report 


14 


17 


Strategic long-term linkages 
with regional partners 


Diagram of governance 


Board composition 


Five-year programme 


Table 1.2, 

The Asia Pacific Finance and 
Development Centre (AFDO), 
China based at the Shanghai 
National Audit Institute 





WB oversight functions 


Implicit theory of change 


While CLEAR was originally 
intended as an experiment or 
pilot, this has not been 
strongly reflected in how the 
initiative was designed or 
managed. 


Theory of Change 


This statement presumes that long-term linkages with 
regional partners are essential to the strategies to be 
followed by every center, which is not the case. 


The RACs need nat include all of the different 
stakeholder groups displayed in the diagram. The RAC 
composition is determined by the centers and not based 
on stakeholder groupings. The RAC ToRs were modified 
in 2013 


Technically, the Board is composed of donors providing 
$500K or more, so not all agencies are full Board 
members. (Please see the Board ToRs) 


It would be more accurate to say that “...the centers 
were established at different times as and when funding 
became available...” The statement as it is does not 
provide the full picture of why the centers were 
established at different times. 


Shanghai National Accounting Institute Factual error corrected in final report 


The functions are not simply oversight; they are also to 
provide technical support and guidance. 


The evaluation discusses using an implicit theory of 
change. Could this be captured as part of the 
methodology section? 


CLEAR encompasses a learning-by-doing model. lt was 
not conceived of as a formal experiment with pre- 
specified hypotheses but rather as an experiment in 
broader terms. 


The theory of change is generic at the program level; it 
does not imply that the specificities of it would not be 


| Page No. | Issue 0 Correction/key comment/clarification Response to key comment! 


different depending on the center. Each centers 
strategy takes into account the different elements of the 
ToC -- how it defines the different outcome areas and 
the stakeholder groups it works with (implicitly or 
explicitly). This is precisely the reason that that there 
are differences in how the concepts are operationalized. 
The operationalization of the program was not intended 
to be uniform across all centers, precisely because it is 
context driven and tailored to regional circumstances 


and realities. 
18 Discussion of results Please see comment with respect to page iii (measuring 
framework “success”) 
22 Annual report lacks focus on It would be helpful if the evaluation could clarify why the 
outcomes discussion in the 2013 annual report on how the centers 


are contributing to the outcomes outlined in the results 
framework is not considered to be progress toward or 
contributing to outcomes. What other type of evidence is 
needed for progress toward outcomes? 


23 IEG was “selected” This statement implies a conscious selection process. Rephrased to address comment 
IEG established the trust fund and the partnership and 
was therefore the default office for the secretariat. It 
would be more accurate simply to say that IEG houses 
the Secretariat. 


23 Secretariat workload It would be more accurate to say that the Secretariat Rephrased to address comment 
“workprogram” rather than “workload” (the latter is not is 
reviewed by the Board). Once there was a discussion 
on the workload, but this has not been an ongoing Board 
concern. 


28 Secretariat It would be more accurate to describe the Secretariat's Rephrased to address comment 
work not just as administration but also overall 
management, including operational strategy. 


24 Reference to leadership It would be helpful if the evaluation could elaborate on 
what is meant by “gaps in leadership” — is it to address 
the questions outlined on page 25? 


27 Contexts not sufficiently It would be helpful to understand how contexts could be 
reflected in program design reflected better, beyond what the program did: conduct 


| Page No. | Issue = sid Correction/key comment/clarification Response to key comment! 


28 


30 


34 


Focusing on delivering vs. 
learning 


Centers have had few 
opportunities to elicit inputs 
from regional experts 


Several centres experienced 
delays and challenges due to 
difficulties in aligning host 
institution rules and 
procedures with World Bank 
requirements. In several 
cases (especially in AA and 
FA) this led not only to delays 
in the transfer of funds to the 
centres, but also to inefficient 
use of professional staff, 
given that they were tied up 
with administrative tasks. 


preparatory studies in the CLEAR regions and use a 
competitive process requiring the shortlisted institutions 
to provide information on the regions needs/demands 
and a strategy to address those needs/demands. 


The program’s focus is on learning by doing (not either 
delivery or learning), although less than desirable 
emphasis has been given to learning 


This statement is misleading because centers elicit Rephrased paragraph to address 
inputs and views from a wide range of experts informally, comment 
which does not get reflected in formal documents. 


The delays were problematic not only for the centers but 
also for the Secretariat. WB procurement and related 
replies were timely, but often the documents were 
delayed more with the host institutions or host 
governments. 


AA comments A 


35 


A first grant agreement 
between the World Bank and 
Wits was signed In February 
2012, for a grant of USD 
940,513 for the period June 
2011- January 2013. A 
second grant agreement for 
USD 2,997, 325 was signed 
in June 2014 for the period 
2014-2018. 


The first grant was signed by the World Bank in Corrected in final version 


December 2011 and the second grant in May 2014. 


AAA Correction/key comment/clarification Response to key comment! 


35 


38 


39 


47 


Thus the AA centre has 
effectively had no grant 
support for ongoing activities, 
besides resources it has 
raised on its own. 


Note 69 -While the grant was 
signed by both parties in 
February 2012, the World 
Bank considers June 2011 as 
the start-up date. 


The 2013 draft strategy has 
not yet been fully applied due 
to long delays in receiving the 
World Bank grant as well as 
current gaps in AA centre 
leadership and capacities 


Note 78 - While the CLEAR 
Secretariat indicates that 
Centres were free to develop 
their own theories of change, 
the AA centre reports that it 
was told to incorporate the 
overall ToC into its strategy 
document as it moved 
towards strategy approval by 
the CLEAR Board” 


While the World Bank 
committed funding to the 


The funds for the CLEAR Center have been in the Trust 
Fund. The lack of access to grant funds is related to: 

- Delays from the Center in turning ina 
Project Document that had a structure and 
content that were ready to go through World 
Bank clearances. 

- Delays from Wits in signing the grant, 
providing information on the authorized 
signatories, providing banking information 
and registering in the World Bank Systems 
(client connection) to be able to access the 
funds. 


Retroactive financing that covered activities from May 
2011 onwards, meaning that the center could claim 
expenses backdating to May 2011. 


The 2013 draft strategy has not yet been fully applied 
due to long delays in being able to finalize the 
procedures necessary to receive the World Bank grant 
as well as current gaps in AA centre leadership and 
capacities” 


The Secretariat suggested using the program’s ToC 
because the Center had not yet envisioned a ToC on 
their own and there were already significant delays with 
the completion of the Project Document, which cause 
many of the issues highlighted in the evaluation. 


(General comment not limited to AA) 


Rephrased to address comment, and 
shifted paragraph to other location in 
chapter to ensure better logical flow. 


Reflected in revised report- 


Rephrased to address comment 


Rephrased to address comment 


Corrected throughout the report. 
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each of the CLEAR Regional 
Centres for a five-year period, 
the initial contract with the AA 
centre was for one year, and 
subsequent funding was 
linked to satisfactory 
performance.89 While this 
might have addressed the 
World Bank’s need to 
manage risks, it appears 
incompatible in the context of 
an experimental initiative 
such as CLEAR which 
naturally would take time to 
gestate. 


The establishment of the AA 
centre was adversely affected 
by delays in finalising the 
original contract between 
Wits and the World Bank due 
to various due diligence 
procedures and other 
requirements of both 
institutions. 


Throughout the report there are references to the “World 
Bank funding” (and related phrasing). It would be better 
to say “multi-donor CLEAR funding administered through 
the World Bank”. 


We also suggest adding a sentence early in the report 
as follows (please contact Maurya in the Secretariat on 


Added suggested sentence in section 
any questions related to this text). E 


CLEAR grants to centers are administered through the 
World Bank's administrative, procurement, financial, 

legal and related arrangements. The grants use funds 
from the multi-donor trust fund established for CLEAR. 


It was the CLEAR Initiative that committed funding for 5 
years, not the World Bank. The trust fund from which 
the grants are provided cannot make commitment of 
funds that it does not hold in cash. At the time the first 
grant was provided to AA, the TF did not hold sufficient 
funds to cover a longer period of time. For the second 
grant, the donors had already deposited additional funds 
into the TF account, so it was possible to make a 3 year 
grant. 


Rephrased to address comment 


These issues are related to managing risks on behalf of 
the donors, since the funds are held in trust. 


Please see comments above, under page 35, about 
World Bank's role in the delays. 


Rephrased to address comment 


E AS Correction/key comment/clarification Response to key comment! 
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The centre also experienced 
delays in identifying and 
appointing a Centre Director 
acceptable to the World 
Bank, and in preparing and 
approving a project contract 
for the period 2013-18. 


The nature of the agreements 
between the World Bank and 
the AA centre (as well as the 
AA centre and Affiliate 
Centres) led to long delays 
between the conduct of an 
activity and reimbursement. 
(...) Interviews suggest that 
in some instances 
reimbursements could take 
up to 5-6 months for 
completed pre-approved 
programmes. 


The centre also experienced delays in identifying and 
appointing a Centre Director acceptable to the initiative's 
Board, not the World Bank. 


Rephrased to address comment 


Reimbursements are made within a week or two by the 
world Bank. The delays were related to Wits taking time 
in submitting the paperwork to be reimbursed. 


Rephrased to address comment 


LA Comments 


58, 59 


Delays in the processing of 
the CLEAR grant did not 
negatively affect centre 
activities, as the CLEAR LA 
centre was able to access 
funding from other sources. 
(...) The first CLEAR grant of 
USD 595,000 awarded to the 
LA centre in May 2013 was 
received in May 2014. Delays 
were due to a number of 
factors, including the need to 
reconcile policies and 
procedures of the CLEAR 
Secretariat, the Mexican 
government, and CIDE. “ 


These statements seem to imply that there were delays 
from the World Bank side. The grant was signed at the 
World Bank on May 13, 2013 and the funds were 
available from that point on (not in November 2013, as 
the evaluation notes). The delays were related to: 


- Internal procedures at CIDE to comply with 
World Bank procurement and FM 
requirements 
- Delays in the signature of the grant by the 
Government of Mexico (The administration 
had just changed and the head of the office 
in charge of these signatures was not 
appointed) 
Also, the fact that CIDE was able to leverage funds from 
other donors did not cause the delays. It was the other 
way around. It was actually getting these funds that got 


Rephrased to address comment 
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“The grant seemed ready to 
be disbursed in November 
2013, but at that point the LA 
centre asked the Secretariat 
to postpone disbursement to 
early 2014 as receiving the 
grant late in the year would 
have posed considerable 
administrative challenges. 
The process took another 
four months and funds were 
received in May of the 
following year. In the 
meantime, major donors that 
the centre was able to 
leverage were the Mexican 
Government106 (USD 
500,000) and the Bill and 
Melinda Gates Foundation 
grant (USD 186,395)” 


CIDE to delay processing the grant, since the funds that 
they leveraged had to be disbursed quickly. Also, they 
wanted avoid having to deal with administrative 
processes (reporting, audits, etc.) if they were not going 
to use the funds. 


Leveraging funds was not a strategy to compensate for 
delays in the grant. It was a purposeful decision to 
attract more funds and expand the program. CIDE would 
have not been able to leverage the extra funds without 
signing the agreement with the Bank. And would have 
not been able to spend the funds it had leveraged unless 
it delayed the expenditures from the World Bank grant. 


65 


National M&E Office 
Threshold of a “large project” 


IEO was disbanded September 2014. 


Dollar thresholds of projects requiring clearance is 
provided in the grant agreements. The approvals had to 
do with the nature of the contracting — whether to do a 
sole-source or a competitive bid, in order to follow 
procurement rules. 


Corrected 
Added this information 


The East Asia centre is the 
only CLEAR centre whose 
host institution was not 
selected via a competitive 
process. Instead, the Asia 
Pacific Finance and 
Development Centre (AFDC) 
in China based at the 
Shanghai National Audit 


Suggested rephrasing: 

The CLEAR East Asia centre — based at the Asia Pacific 
Finance and Development Centre (AFDC) in China 
based at the Shanghai National Accounting Institute 
(SNAI) - is the only of the currently existing centres 
whose host institution was not selected via a competitive 
process. Before CLEAR was developed, the World 
Bank, Asian Development Bank, and other partners in 
2007 came together to launch the Shanghai International 


Rephrased according to suggestion 


72 


72 


Institute (SNAI) was selected 
to host the EA centre based 
on its existing relationship 
with the World Bank around 
hosting and facilitating the 
Shanghai International 
Program in Development 
Evaluation Training 
(SHIPDET) since 2007 (see 
sidebar). 


To date, the EA centre has 
offered two kinds of regular 
training programmes: 
SHIPDET and training on 
Impact Evaluation. In 
addition, it has offered ad hoc 
training on Performance 
Budgeting. As shown in the 
sidebar, the centre has 
engaged in a number of 
advisory services, knowledge 
exchange activities, as well 
as internal capacity building 
efforts for its staff. To date, it 
has developed one 
knowledge resource. 


The centre does not have 
confirmed data on the 
composition of its clients, but 
estimates that 50% of clients 
are government officials from 
different countries, 20% are 
from academia, while the 
remaining 30% are evaluation 
practitioners, including 
independent consultants, as 
well as staff from auditing and 
evaluating consulting firms. 


Correction/key comment/clarification Response to key comment! 


Program in Development Evaluation Training 
(SHIPDET). SHIPDET has enjoyed success in training 
evaluation practitioners, disseminating evaluation 
knowledge, and networking, not only in China, but also 
in the whole region. When the CLEAR program was 
begun to further promote learning on evaluation and 
results, AFDC was selected to host the EA centre based 
on its existing relationship with the World Bank around 
hosting and facilitating SHIPDET since 2007. 


Suggested adding at the end of the paragraph. Added according to suggestion 


In addition to the core course deliveries, like IPDET and 
Impact Evaluation, the SHIPDET program also has 
specialized courses (usually of two to three days) on a 
range of topics. Recent specialized topics include 
Performance Based Budgeting, Introduction to Impact 
Evaluation, Cost-Benefit Analysis for Evaluators, and the 
Case Study Approach to Evaluation. 


Zhao Min from AFDC will send exact data. 
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US 


SHIPDET 


Finding 52. The 
establishment of the EA 
centre took place ina 
regional environment 
characterised by national 
governments (in particular 
China) increasingly pushing 
for and establishing systems 
for M&E and performance- 
based budgeting. The supply 
of quality RBM/M&E services 
was scarce, and awareness 
of the benefits of high-quality 
M&E and RBM not yet wide 
spread or institutionalised. 
This context has not 
significantly changed. 


Finding 54. The EA centre, 
through its host institution 
AFDC, possesses strong 
managerial capacity. The 
centre’s current professional 
capacity is weak and the 
delivery of capacity building 
services is reliant ona 
network of international 
resources. 


Finding 57: The EA centre is 
building strong partnerships 
with the government of China 
and with public finance 
institutions in the region. It 
has not proactively reached 
out to other development 
actors, including multilateral 


While the finding is not incorrect, it underemphasizes the 
progress that the center has made in awareness raising 
and governmental changes within the region, and most 
especially with MOF in China. We request that the last 
paragraph in the narrative section be captured more in 
the finding. The EA center can provide more information 
on the influence that they have had (e.g., Recently as a 
result of MOF staff taking SHIPDET training, a 
Performance Based Budgeting Committee has been set 
up in China with MOF heading it and AFDC having the 
Deputy Chair role in this committee. The committee 
could be considered a quasi- evaluation association 
within a government setting. This is a direct result of the 
SHIPDET-CLEAR program on SHPDET and PBB and is 
an example of a result that could be highlighted. The 
AFDC team can provide more detailed/accurate 
narrative on this.) 


While the overall finding and supporting notes are 
correct, it would be useful to emphasize more both (a) 
the nature of the AFDC staff trainers’ work on the 
courses for China participants (in Chinese) with the staff 
taking over more teaching duties, and (b) the internal 
capacity building that the AFDC staff have done —i.e., 
participating in the IPDET program in Canada, attending 
international evaluation association conferences — to be 
able to build their knowledge and skills to take on more 
teaching duties. 


The center has worked in partnership with WB, ADB, 
IFAD, UNDP, and 3ie. It is also exploring closer 
connections to evaluation associations in the greater 
regions and other actors. At roughly 2+ years in the 
program, the center is at a point where they are going to 
be stepping up their outreach/engagement. Though it is 
noted that the center has been somewhat conservative 
in its outreach approach — wanting to gain sound footing 
before reaching out too soon. 
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and bilateral agencies, CSOs 
and academia. 


Finding 59: The EA centre 
has established a Regional 
Advisory Committee, but it is 
not yet operational. This has 
diminished the centre’s ability 
to validate programming 
plans and decisions through 
a broader stakeholder group. 


Resources from the CLEAR 
grant have been used almost 
exclusively for client capacity 
building activities, with only a 
small amount being used for 
internal capacity building of 
EA centre staff. The majority 
of activities conducted by the 
EA centre to date have, 
however, been paid for 
through other sources.134 
The Chinese MoF has been 
supporting AFDC and thereby 
also CLEAR operation costs 
(mainly staff salaries) and 
has been sponsoring the 
participation of Chinese 
participants in various 
CLEAR training courses. So 
far, the Chinese government 
has contributed at least USD 
3.5 million to the CLEAR EA 
programme. 


Integration in and support 
from host institution: While 
AFDC has provided ongoing 
managerial, administrative, 
and professional support to 


Please see other commentary on the overall CLEAR 
RAC arrangements in this note. At the same time, we 
agree that the program would benefit from having more 
non-Chinese participation in the RAC to advise the 
center on non-China opportunities/connections. We also 
note that the center has excellent arrangements within 
China related to the RAC and government connections. 


Suggest adding more emphasis to the rather impressive 
contributions that MoF and AFDC have made to the 
program related to in-kind support — especially given the 
relatively low grant amount. It underscores the 
commitment of both AFDC and MoF to both CLEAR and 
evaluation within the region. This importance of this 
commitment gets a little lost in the text. 


The paragraph mixes up a few things. While initially 
SHIPDET was a stand-alone program, it now is run 
under the CLEAR umbrella. Itis fully aligned with 
CLEAR — however the AFDC team recognizes that the 
SHIPDET name had/has established name recognition 


Rephrased to address comment 
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centre activities, it has made 
limited investments in CLEAR 
in terms of allocating senior 
and full time staff resources. 
This limits the centre’s 
capacity to engage in 
strategic (financial and 
programmatic) longer-term 
planning. Also, given that 
most of the centre’s work has 
been linked to SHIPDET, the 
question arises whether the 
additional services and profile 
brought by the CLEAR centre 
provide sufficient incentives 
for AFDC to continue hosting 
the centre (given the 
administrative and reporting 
requirements), or whether it 
might be more beneficial for 
AFDC to go back to merely 
hosting SHIPDET. 


Context: Document review 
and stakeholder consultations 
indicate that there is ample 
demand for M&E capacity 
building services in China as 
well as in the larger East Asia 
region. The Chinese MoF will 
almost certainly continue its 
support for the operation 
costs of AFDC, as well as its 
sponsorship of Chinese 
participants for training 
courses such as SHIPDET, 
regardless of whether these 
are provided through AFDC 
or CLEAR. While the Chinese 


within Asia, so the center has kept the name SHIPDET 
and essentially operated in a co-branding arrangement. 


Would be good to explore with MoF and the AFDC team 
whether the Chinese government is unlikely to be 
interested in investing in the expansion of the EA 
centre’s current work. Hmightbe-betterto say “we're 

if the Chi Sean 


“ ” 


As a side note, the AFDC team has expressed relatively 
fewer complaints about reporting than the other centers. 
So not sure how much they look at reporting as a heavy 
burden as they already have an established norm to 
collect and report the basic data requested. It is noted 
that as a next step, they would benefit from reporting 
more on outcomes/results to highlight achievements in 
the region, but in terms of providing input, activity and 
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government is unlikely to be 
interested in investing in the 
expansion of the EA centre’s 
current work (e.g. deepening 
its regional approach, or 
adding a more distinct 
research component), other 
donors with an interest in the 
East Asia/Pacific region (e.g. 
the ADB or AuSAID) may be 
willing to explore related 
options. 


Overall, there are a number 
of positive factors that can 
support the financial viability 
and continued relevance of 
the CLEAR EA centre (in 
particular the demand for 
M&E capacity building and 
the availability of likely 
sources of funding). The 
bigger question, however, is 
whether the benefits of 
hosting a CLEAR centre (in 
terms of financial resources, 
contacts/networking 
opportunities, reputational 
affiliations) outweigh the 
drawbacks for AFDC (e.g. 
added reporting obligations 
and administration). 


AFDC has to decide whether 


(or under what conditions) 
hosting CLEAR continues to 
make sense for AFDC. 


output information, it doesn’t seem to be a terrible 
burden for AFDC. 


Suggest revisiting this as AFDC, we believe, has not 
questioned whether it makes sense to host CLEAR. All 
other findings seem reasonable/good. 


Co Correction/key comment/clarification Response to key comment! 
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Initially the centre was 
operating with one full-time 
person (the coordinator), one 
part-time training specialist, 
one financial officer who later 
left the centre, and a part- 
time procurement officer. 


An intern joined the team in 
2013 


Capacity development 
activities conducted to date 
appear to have been guided 
by the broader objective to 
establish the centre as a 
provider of customised M&E 
training and capacity 
development services with a 
focus on strengthening the 
supply side of M&E. 


While the demand study 
indicated that the centre’s 
work during its first year 
would place particular 
emphasis on the three 
countries covered in the 
study, this has only been 
visible for Senegal, but not for 
Benin and Mauritania. 


The centre has not yet 
developed an explicit (or 
indicated the existence of an 
implicit) regional or country- 
specific Theory of Change. 
The complexity of regulations 
around the CLEAR grant has 
slowed down the payment of 
accounts. (Most consultants 


Financial officer had to leave due to unsatisfactory 
performance. Currently, there is a financial specialist 
(who oversees all the donor-funded programs within 
CESAG) who partially works for CLEAR with support of 
his assistant. A part-time procurement consultant has 
been hired since May 2014. 


An intern officially joined the team in February 2014. 


The Project Document refers not only to 
“training/workshops” but also to “advisory/knowledge 
service, evaluation and applied research’. 


The demand study indicates the potential for CESAG 
M&E offerings to the country; however, this did not mean 
that CESAG planned to place emphasis on all three 
countries at once. The assessment helped rule out 
Mauritania due to its low score on all M&E elements. 


The centre is in the process of developing a regional 
specific ToC together with the Strategic Plan. 


This is due to CESAG’s cumbersome and bureaucratic 
accounting procedure, not because of the regulations 
around the CLEAR grant. 


Addressed in revised version 


Corrected 


Addressed in revised version 
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interviewed said they had 
waited six months or more for 
their accounts to be settled.) 


A number of additional comments that were not included in the table above were provided by the five CLEAR centres and the Board. However, 
with one exception these did not address factual errors. Only the South Asia centre noted a number of corrections required in Volume II, Annexes. 
These corrections were made and are reflected in the revised Volume Il. The evaluation team noted all other comments with thanks 


